When you develop with docker and use bind mounts, you may encounter problems if you do ont align the uid / gid of the user in docker with teh uid / gid of your user. For this purpose, some Dockerfile propose an ARG so that you can pass in the UID/GID you need.
But how to find the right id to give ?
On linux, simply run: id
On windows, according to this stackoverflow contribution you should open a command window and run:
| -- FIRST: backup database before anything !!!!! | |
| -- drop indexes for prediction table for perfs | |
| DROP INDEX prediction_barcode, prediction_data, prediction_server_domain, prediction_source_image, prediction_timestamp, prediction_type; | |
| -- move rows | |
| WITH to_copy_rows AS ( | |
| SELECT | |
| barcode, |
| """ | |
| Get csv from postgresql: | |
| ``` | |
| docker-compose exec postgres psql -W -U postgres postgres | |
| postgres# \copy (select barcode, STRING_AGG(value_tag, ' / ') from product_insight where automatic_processing = 't' and timestamp > '2022-01-20' and type = 'category' group by barcode order by barcode) to '/tmp/auto-cat.csv' with csv; | |
| COPY 13910 | |
| ``` | |
| Run this below: | |
| ``` |
| import collections | |
| import statistics | |
| def run_exp(k, N): | |
| """run one experimentation, that is stream selecting k items among N""" | |
| result = [] | |
| seen = 0 | |
| for i in range(N): | |
| seen += 1 | |
| index = random.randrange(seen) |
I had a corrupted tar file (due to interruption while writing last file …).
I needed to append to archive and it was not possible (I had tarfile.ReadError: empty header as soon as I opened the archive in append mode.
Problem : the archive had 1M°+ files all flat (it was intended to use directly).
This script save my day, and was quite fast (Note how opening the big tar is expensive: 9 minutes !):
2022-05-17T15:52:17.492944 starting
2022-05-17T16:03:01.681669 0 done, 0 errs
2022-05-17T16:05:01.001833 100000 done, 0 errs
| # run it using docker-compose run --rm worker_high_1 python3 | |
| from robotoff.prediction.category.neural.keras_category_classifier_3_0 import * | |
| from robotoff.off import * | |
| from robotoff.models import * | |
| import json | |
| import os | |
| product_id = ProductIdentifier(barcode="3017620422003", server_type=ServerType.off) |