Face attribute dataset creation by automatic image downloading and processing

face-attributes-grouped.ipynb

{"cells":[{"cell_type":"markdown","source":["# Face Attributes Grouped\n","\n","This notebook is the work of collecting **Face Attributes Grouped** dataset that can be accessed [here](https://www.kaggle.com/datasets/mantasu/face-attributes-grouped). It basically shows how to download images by keywords automatically and group them to categories. It also shows how to process the downloaded images to automatically align and center-crop the faces.\n","\n","> **Note**: you may want to check [Face Crop Plus](https://github.com/mantasu/face-crop-plus) and [Image Downloader](https://github.com/QianyanTech/Image-Downloader) repositories.\n","\n","**IMPORTANT:** if this notebook is run on [Google Colab](https://colab.research.google.com/), please change the runtime type to GPU: `Runtime -> Change runtime type` and set _Hardware accelerator_ to _GPU_."],"metadata":{"id":"NmZKQnMiYtKt"}},{"cell_type":"markdown","source":["## Setup Libraries\n","\n","Run the cells below to install the required image processing and image-downloader libraries. Some default code has to be changed to maintain the names of the images downloaded."],"metadata":{"id":"i-6XADuOY-6_"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"1OXhl37wX30T"},"outputs":[],"source":["!pip install -q face-crop-plus selenium chromedriver_autoinstaller\n","!git clone https://github.com/QianyanTech/Image-Downloader.git\n","%cd Image-Downloader"]},{"cell_type":"code","source":["def replace_line(filepath, old_line, new_string):\n","    with open(filepath, 'r') as file:\n","        data = file.readlines()\n","    for i in range(len(data)):\n","        if data[i].strip() == old_line:\n","            data[i] = new_string + '\\n'\n","    with open(filepath, 'w') as file:\n","        file.writelines(data)"],"metadata":{"id":"iofdKPm11Z6r"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["replace_dict = {\n","    'import os': 'import os\\nimport re',\n","    'new_file_name = \"{}.{}\".format(file_name, file_type)': '                new_file_name = file_name',\n","    'file_name = file_prefix + \"_\" + \"%04d\" % count': '            if (mtch := re.search(r\"\\/([\\w-]+\\.(?:jpg|jpeg|png|gif))\", image_url)) is not None:\\n                file_name = mtch.group(1)\\n            else:\\n                file_name = image_url'\n","}\n","\n","for old_line, new_string in replace_dict.items():\n","    replace_line(\"downloader.py\", old_line, new_string)"],"metadata":{"id":"hnQYvmjm1sua"},"execution_count":null,"outputs":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"z4zzK6WZw8i6"},"outputs":[],"source":["import os\n","import shutil\n","from tqdm import tqdm\n","from crawler import crawl_image_urls\n","from downloader import download_images\n","from utils import resolve_dependencies\n","from face_crop_plus import Cropper"]},{"cell_type":"code","source":["def download_from_keywords(keywords: str, subdir: str = \"../images\"):\n","    proxy, proxy_type = None, None\n","\n","    engine = \"Bing\"\n","    driver = \"api\"\n","    max_number = 1000\n","    num_threads = 40\n","    timeout = 10\n","    output = subdir\n","\n","    if not resolve_dependencies(driver):\n","        print(\"Dependencies not resolved, exit.\")\n","        return\n","\n","    crawled_urls = crawl_image_urls(\n","        keywords,\n","        engine=engine,\n","        max_number=max_number,\n","        browser=driver,\n","        face_only=False,\n","        safe_mode=False,\n","        proxy_type=None, \n","        proxy=None,\n","        image_type=None,\n","        color=None\n","    )\n","\n","    download_images(\n","      image_urls=crawled_urls,\n","      dst_dir=output,\n","      concurrency=num_threads,\n","      timeout=timeout,\n","      proxy_type=None, \n","      proxy=None,\n","      file_prefix=None\n","    )\n","\n","    print(\"Finished.\")"],"metadata":{"id":"fczu1RZnbIQu"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Image Downloading\n","\n","Here, the target download group is specified (`headwear` is taken as an example). The search keywords for each group are included in the further cell. Some sub-categories require more keywords than others for as it is very difficult to find the specific full-face images for some attributes. This section may take a while to download all images for all the desired search queries."],"metadata":{"id":"8Ce84LB0ZRN4"}},{"cell_type":"code","source":["target_dir = \"headwear\""],"metadata":{"id":"BavkDwRQW9pM"},"execution_count":null,"outputs":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ci8orW8NVs7E"},"outputs":[],"source":["KEYWORD_DICT = {\n","    \"headwear\": {\n","        \"hoodie\": [\n","            \"person wearing black hoodie\", \"person wearing colorful hoodie\", \"man wearing green hoodie\", \"man wearing blue hoodie\", \"man wearing yellow hoodie\", \"man waring red hoodie\", \"woman wearing green hoodie\", \n","            \"woman wearing blue hoodie\", \"woman wearing yellow hoodie\", \"woman waring red hoodie\", \"woman wearing cloth on head\", \"wearing scarf on head\", \"head covered under blanket\", \"wearing islamic hoodie\", \n","            \"head covered under hoodie\", \"head covered under cloth\", \"person wearing raincoat hoodie\", \"man wearing raincoat hoodie\", \"woman wearing raincoat hoodie\", \"person wearing jacket hoodie\", \"man with jacket hoodie\", \n","            \"woman with jacket hoodie\", \"man with white hoodie on head\", \"man with black hoodie on head\", \"man with red hoodie on head\", \"man with green hoodie on head\", \"man with blue hoodie on head\", \"man with yellow hoodie on head\", \n","            \"man with pink hoodie on head\", \"woman with white hoodie on head\", \"woman with black hoodie on head\", \"woman with red hoodie on head\", \"woman with green hoodie on head\", \"woman with blue hoodie on head\", \n","            \"woman with yellow hoodie on head\", \"woman with pink hoodie on head\", \"woman with cat hoodie on head\", \"man with cat hoodie on head\", \"person with hacker hoodie on head\", \"person with assassin's creed hoodie\", \n","            \"man with robe hoodie on head\", \"woman with robe hoodie on head\", \"man with hoodie shining in dark\", \"woman with hoodie shining in dark\", \"person with glowing hoodie on head\", \"person with hoodie no face can be seen\",\n","            \"wearing spiderman hoodie on head\", \"wearing red riding hoodie on head\", \"wearing spooky hoodie on head\", \"wearing funny hoodie on head\", \"wearing girly hodie on head\", \"woman wearing pikachu hoodie\", \"woman wearing wold hoodie\", \n","            \"woman wearing dog hoodie\", \"woman wearing bunny hoodie\", \"man wearing pikachu hoodie\", \"man wearing wold hoodie\", \"man wearing dog hoodie\", \"man wearing bunny hoodie\"\n","        ],\n","        \"helmet\": [\n","            \"wearing astronaut helmet\", \"wearing military helmet\", \"wearing soldier helmet\", \"wearing builder helmet\", \"wearing structure worker helmet\", \"wearing cycler helmet\", \"wearing ice hockey helmet\", \n","            \"wearing american football helmet\", \"wearing firefighter helmet\", \"wearing ski helmet\", \"wearing scienticst helmet\", \"wearing motorcycle helmet\", \"wearing diver helmet\", \"wearing jet helmet\", \n","            \"wearing SWAT helmet\"\n","        ],\n","        \"top\": [\n","            \"wearing tophat\", \"wearing towel on head\", \"wearing sunhat\", \"wearing cowboy hat\", \"wearing cap\", \"wearing hat\", \"wearing turban\", \"wearing beanie\", \"wearing beret\", \"wearing crown\", \"wearing baseball cap\", \n","            \"wearing visor\", \"wearing headband\", \"wearing horns\", \"wearing animal ears\", \"wearing headband\"\n","        ],\n","    },\n","    \"eyewear\": {\n","        \"eyeglasses\": [\n","            \"man wearing round glasses\", \"woman wearing round glasses\", \"wearing cat-eye glasses\", \"wearing square glasses\", \"man with spectacles\", \"woman with spectacles\", \"woman with eyeglasses\", \"wearing horn-rimmed glasses\", \n","            \"wearing browline glasses\", \"wearing full-rimmed glasses\", \"wearing half-rimmed glasses\", \"wearing lab glasses\", \"wearing glasses without frames\", \"wearing monocle\", \"wearing panto glasses\", \"wearing swimming googles\"\n","        ],\n","        \"sunglasses\": [\n","            \"wearing pinhole glasses\", \"wearing party sunglasses\", \"wearing colored glasses\", \"wearing clubmaster sunglasses\", \"wearing wayfarer sunglasses\", \"wearing aviator sunglasses\", \"wearing navigator sunglasses\", \n","            \"man with colored sunglasses\", \"woman with colored sunglasses\", \"wearing butterfly sunglasses\", \"wearing square sunglasses\", \"wearing round sunglasses\", \"wearing rimless sunglasses\", \"wearing MLG sunglasses\", \n","            \"wearing 3d glasses\", \"wearing ski glasses\"\n","        ],\n","        \"eyecover\": [\n","            \"wearing eye patch\", \"wearing blindfold\", \"wearing vr headset\", \"wearing sleeping eyecover\", \"wearing eye shield\", \"wearing eye pillow\", \"wearing eye mask\", \"wearing eye compress\", \"wearing eye bandage\", \"wearing eye visor\", \n","            \"wearing ninja eye mask\", \"person with covered eyes\", \"person with cat eyemask\", \"person with carnival eyemask\", \"wearing eye cover for dry eyes\", \"wearing pinata eye cover\", \"man with cucumbers on face\", \"woman with cucumbers on face\",\n","            \"wearing blue blindfold\", \"wearing red blindfold\", \"wearing green blindfold\", \"wearing cat eye mask\", \"wearing bat eyemask\"\n","        ],\n","    },\n","    \"facewear\": {\n","        \"mouthmask\": [\n","            \"wearing surgical mask\", \"wearing respirator\", \"mouth covered with cloth\", \"mouth covered with scarf\", \"mouth covered with towel\", \"mouth covered with tape\", \"wearing N95 respirator\", \"wearing KN95 respirator\", \"wearing beek\", \n","            \"wearing FFP2 respirator\", \"wearing funny facemask\", \"wearing mouth mask with zip\", \"wearing colored facemask\", \"wearing dust mask\", \"wearing handmade mask\"\n","        ],\n","        \"fullmask\": [\n","            \"wearing gasmask\", \"wearing welding mask\", \"wearing escape mask\", \"wearing halloween mask\", \"wearing bandit mask\", \"wearing burglar mask\", \"wearing anonymous mask\", \"islamic woman facecover\", \"wearing thermal mask\", \n","            \"wearing carnival mask\", \"wearing animal face mask\", \"exfoliating mask\", \"clay mask\", \"gel mask\", \"sheet mask\", \"wearing cream mask\", \"man with red mask\", \"woman with red mask\", \"man with blue mask\", \"woman with blue mask\", \n","            \"man with green mask\", \"woman with green mask\", \"man with fetching mask\", \"woman with fetching mask\", \"person with saw mask\", \"person with squid game mask\", \"person with troll face mask\", \"man with disney mask\", \n","            \"woman with disney mask\", \"person with lion mask\", \"person with bear mask\", \"person with wolf mask\", \"person with spiderman mask\", \"person with deadpool mask\", \"person with marvel mask\", \"person with dc mask\",\n","            \"wearing iron man mask\", \"wearing hulk mask\", \"wearing thor mask\", \"wearing batman mask\", \"wearing spiderman mask\", \"wearing flash mask\", \"wearing back panther mask\", \"wearing ant-man mask\", \"black panther\", \"venom spider man\", \n","            \"wearing handmade halloween mask\", \"wearing glowing mask\", \"wearing captain america mask\", \"wearing male clown mask\", \"wearing female clown mask\", \"wearing alien mask\", \"wearing star wars mask\", \"wearing guardians of the galaxy mask\", \n","            \"wearing avatar mask\", \"wearing furry mask\", \"wearing marshmellow mask\"\n","        ],\n","        \"covered\": [\n","            \"faces covered in bubbles\", \"face covered in leaves\", \"face covered in flowers\", \"face covered in petals\", \"face covered with hands\", \"face covered in shadow\", \"man eating\", \"man drinking\", \"woman eating\", \"woman drnking\", \n","            \"covered face\", \"face covered with book\", \"bright light on face\", \"face covered under lights\", \"face covered under butterflies\", \"face hit with ball\", \"woman drinking from huge cup\", \"man drinking with huge cup\", \n","            \"woman face covered with hands\", \"man face covered with hands\", \"face covered with grass\", \"book covering face\", \"person holding book in front of face\", \"person face covered with sign\", \"holding hand in front of face\", \n","            \"dark shadow on face\", \"face covered by plant\", \"face covered by tree\", \"man hiding face\", \"woman hiding face\", \"face emerging from shadows\", \"laptop covering face\", \"computer covering face\", \"microphone covering face\", \n","            \"microphone in front of face\", \"person behind bars\", \"person in jail\", \"face behind bars\", \"woman behind bars\", \"man behind bars\", \"face behind wall\", \"face covered with hair\", \"face behind fence\", \"person behind iron bars\", \n","            \"person behind wooden door\", \"man behind fence\", \"woman behind fence\", \"face inside bush\", \"face behind trap door\", \"face covered with phone\", \"man taking a picture\", \"woman taking a picture\", \"person taking picture with phone\", \n","            \"person taking picture with camera\"\n","        ],\n","    },\n","    \"accessories\": {\n","        \"earrings\": [\n","            \"man with big round earrings\", \"man with square earrings\", \"man with earring in one ear\", \"wearing stud earrings\", \"person with drop earring\", \"person with cluster earring\", \"person with dangle earring\", \"person with hoop earring\", \n","            \"person with huggie earring\", \"person with chandelier earring\", \"person with threader earring\", \"person with barbell earring\", \"person with ear cuffs\", \"person with tassel earrings\", \"person with ear jackets\", \n","            \"man with amber earring\", \"woman with amber earrings\", \"man with blue earrings\", \"woman with blue earrings\", \"man with red earrings\", \"woman with red earrings\", \"man with green earrings\", \"woman with green earrings\", \n","            \"man with yellow earrings\", \"woman with yellow earrings\", \"man with gold earrings\", \"woman with gold earrings\", \"man with amber necklace\", \"woman with amber necklace\", \"man with blue necklace\", \"woman with blue necklace\", \n","            \"man with red necklace\", \"woman with red necklace\", \"man with green necklace\", \"woman with green necklace\", \"man with yellow necklace\", \"woman with yellow necklace\", \"man with gold necklace\", \"woman with gold necklace\", \n","            \"asian with orange earrings\", \"asian with purple earrings\", \"asian with triangle earrings\", \"asian with pink earrings\", \"asian with violet earrings\", \"black person with orange earrings\", \"black person with purple earrings\", \n","            \"black person with triangle earrings\", \"black person with pink earrings\", \"black person with violet earrings\", \"man with orange earrings\", \"man with purple earrings\", \"man with triangle earrings\", \"man with pink earrings\", \n","            \"man with violet earrings\", \"woman with orange earrings\", \"woman with purple earrings\", \"woman with triangle earrings\", \"woman with pink earrings\", \"woman with violet earrings\", \"person with Paraiba Tourmaline earrings\", \n","            \"person with Painite earrings\", \"person with Aquamarine earrings\", \"person with Diamond earrings\", \"person with Black Opals earrings\", \"person with Tanzanite earrings\", \"person with Larimar earrings\", \n","            \"person with Grandiderite earrings\", \"person with Benitoite earrings\", \"person with Red Beryl earrings\", \"person with Alexandrite earrings\", \"person with Fluorite earrings\", \"person with Taaffeite earrings\", \n","            \"person with Sapphire earrings\", \"person with Jeremejevite earrings\", \"person with Ruby earrings\", \"person with Pearl earrings\", \"person with Topaz earrings\", \"person with Seraphinite earrings\", \"person with Moldavite earrings\",\n","            \"man with gowing earrings\", \"woman with glowing earrings\", \"man with small round earrings\", \"woman with small square earrings\", \"man with small round earrings\", \"woman with small square earrings\", \"man with stone earrings\", \n","            \"woman with stone earrings\", \"man with wooden earrings\", \"woman with wooden earrings\", \"man with metal earrings\", \"woman with metal earrings\", \"man with cross earring\", \"woman with cross earring\"\n","        ],\n","        \"necklace\": [\n","            \"person with pendant necklace\", \"person with choker necklace\", \"person with collar necklace\", \"person with bib necklace\", \"person with lariat necklace\", \"person with opera necklace\", \"person with matinee necklace\", \n","            \"person with princess necklace\", \"person with rope necklace\", \"person with chain necklace\", \"person with statement necklace\", \"person with charm necklace\", \"person with tassel necklace\", \"person with y-necklace\", \n","            \"person with multi-chain necklace\", \"person with torc necklace\", \"person with beaded necklace\", \"person with amber necklace\", \"man with chain necklace\", \"woman with chain necklace\", \"asian with orange necklace\", \n","            \"asian with purple necklace\", \"asian with triangle necklace\", \"asian with pink necklace\", \"asian with violet necklace\", \"black person with orange necklace\", \"black person with purple necklace\", \"black person with triangle necklace\", \n","            \"black person with pink necklace\", \"black person with violet necklace\", \"man with orange necklace\", \"man with purple necklace\", \"man with triangle necklace\", \"man with pink necklace\", \"man with violet necklace\", \n","            \"woman with orange necklace\", \"woman with purple necklace\", \"woman with triangle necklace\", \"woman with pink necklace\", \"woman with violet necklace\", \"man with Paraiba Tourmaline necklace\", \"man with Painite necklace\", \n","            \"man with Aquamarine necklace\", \"man with Diamond necklace\", \"man with Black Opals necklace\", \"man with Tanzanite necklace\", \"man with Larimar necklace\", \"man with Grandiderite necklace\", \"man with Benitoite necklace\", \n","            \"man with Red Beryl necklace\", \"man with Alexandrite necklace\", \"man with Fluorite necklace\", \"man with Taaffeite necklace\", \"man with Sapphire necklace\", \"man with Jeremejevite necklace\", \"man with Ruby necklace\", \n","            \"man with Pearl necklace\", \"man with Topaz necklace\", \"man with Seraphinite necklace\", \"man with Moldavite necklace\", \"woman with Paraiba Tourmaline necklace\", \"woman with Painite necklace\", \"woman with Aquamarine necklace\", \n","            \"woman with Diamond necklace\", \"woman with Black Opals necklace\", \"woman with Tanzanite necklace\", \"woman with Larimar necklace\", \"woman with Grandiderite necklace\", \"woman with Benitoite necklace\", \"woman with Red Beryl necklace\", \n","            \"woman with Alexandrite necklace\", \"woman with Fluorite necklace\", \"woman with Taaffeite necklace\", \"woman with Sapphire necklace\", \"woman with Jeremejevite necklace\", \"woman with Ruby necklace\", \"woman with Pearl necklace\", \n","            \"woman with Topaz necklace\", \"woman with Seraphinite necklace\", \"woman with Moldavite necklace\", \"man with tie\", \"woman with tie\", \"man with locket necklace\", \"woman with locket necklace\", \"man with shawl necklace\", \n","            \"woman with shawl necklace\", \"man with bandana necklace\", \"woman with bandana necklace\", \"boy with candy necklace\", \"girl with candy necklace\", \"man with rainbow necklace\", \"woman with rainbow necklace\", \"man with glowing necklace\", \n","            \"woman with glowing necklace\", \"man wearing silver necklace\", \"man wearing long necklace\", \"man wearing colorful necklace\", \"man wearing short necklace\", \"man wearing funny necklace\", \"man wearing stone necklace\", \n","            \"man wearing rock necklace\", \"woman wearing silver necklace\", \"woman wearing long necklace\", \"woman wearing colorful necklace\", \"woman wearing short necklace\", \"woman wearing funny necklace\", \"woman wearing stone necklace\", \n","            \"woman wearing rock necklace\", \"man with wooden necklace\", \"woman with wooden necklace\", \"man with metal necklace\", \"woman with metal necklace\", \"man with square necklace\", \"woman with square necklace\", \"man with triangle necklace\", \n","            \"woman with triangle necklace\", \"man with circle necklace\", \"woman with circle necklace\", \"man with moon necklace\", \"woman with moon necklace\", \"man with thick necklace\", \"man with thin necklace\", \"man with weird necklace\", \n","            \"woman with thick necklace\", \"woman with thin necklace\", \"woman with weird necklace\", \"man with tourist necklace\", \"woman with tourist necklace\", \"woman with shiny necklace\", \"man with shiny necklace\", \"man with gray necklace\", \n","            \"woman with gray necklace\"\n","        ],\n","        \"piercings\": [\n","            \"person with eyebrow piercing\", \"person with snake Bite piercing\", \"person with smiley piercing\", \"person with Monroe piercing\", \"person with labret piercing\", \"person with tongue piercing\", \"person with nose piercings\", \n","            \"person with septum piercing\", \"person with anti-eyebrow piercing\", \"person with bridge piercing\", \"person with cheek piercing\", \"person with third eye piercing\", \"person with horizontal eyebrow piercing\", \n","            \"person with high nostril piercing\", \"person with regular nostril piercing\", \"person with surface piercing\", \"person with Ashley piercing\", \"man with red nostril piercing\", \"man with blue nostril piercing\", \n","            \"man with green nostril piercing\", \"man with red eyebrow piercing\", \"man with blue eyebrow piercing\", \"man with green eyebrow piercing\", \"man with red cheek piercing\", \"man with blue cheek piercing\", \n","            \"man with green cheek piercing\", \"woman with red nostril piercing\", \"woman with blue nostril piercing\", \"woman with green nostril piercing\", \"woman with red eyebrow piercing\", \"woman with blue eyebrow piercing\", \n","            \"woman with green eyebrow piercing\", \"woman with red cheek piercing\", \"woman with blue cheek piercing\", \"woman with green cheek piercing\", \"man with tongue piercing\", \"woman with tongue piercing\", \n","            \"man with gold cheek piercing\", \"woman with gold cheek piercing\"\n","        ],\n","    },\n","    \"natural\": {\n","        \"plain\": [\n","            \"black woman\", \"black man\", \"white woman\", \"white man\", \"asian woman\", \"asian man\", \"woman with no makeup\", \"male\", \"female\", \"male regular face\", \"female regular face\", \"man shaved\", \"man no facial hair\", \"casual female\", \n","            \"typical male\", \"typical female\", \"profile view male\", \"profile view female\", \"woman before makeup\", \"athlete woman\", \"sports woman\", \"athlete man\", \"female swimming athlete\", \"female baseball athlete\", \"female basketball athlete\", \n","            \"female football athlete\", \"female boxing athlete\", \"man basketball athlete\"\n","        ],\n","        \"facepaint\": [\n","            \"man with makeup\", \"man with lipstick\", \"man with eyeliner\", \"man with eyeshadows\", \"woman with makeup\", \"woman with lipstick\", \"woman with eyeliner\", \"woman with eyeshadows\", \"person with extra makeup eyelashes\",\n","            \"person with mascara\", \"person with lip liner pencil\", \"person with lip gloss\", \"person with halloween makeup\", \"person with glitter on face\", \"person with very fncy makeup\", \"man with green lipstick\", \"man with blue lipstick\", \n","            \"woman with green lipstick\", \"woman with blue lipstick\"\n","        ],\n","        \"facialhair\": [\n","            \"person with Clean-shaven\", \"person with Stubble\", \"person with Goatee\", \"person with Van Dyke\", \"person with Circle beard\", \"person with Soul patch\", \"person with Mutton chops\", \"person with Chinstrap beard\",\n","            \"person with Horseshoe mustache\", \"person with Handlebar mustache\", \"person with Chevron mustache\", \"person with Walrus mustache\", \"person with Fu Manchu mustache\", \"person with Pencil mustache\", \"person with Dali mustache\",\n","            \"person with English mustache\", \"woman with mustache\", \"woman with beard\"\n","        ],\n","        \"facemarks\": [\n","            \"person with pimples\", \"person with freckles\", \"person with scars\", \"person with wrinkles\", \"person with age spots\", \"person with face wounds\", \"person with moles\", \"person with birthmarks\", \"person with acne scars\", \n","            \"person with rosacea\", \"person with eczema\", \"person with psoriasis\", \"person with vitiligo\", \"person with melasma\", \"person with birth face marks\", \"man with face scars\", \"man with wounded face\", \"man with big mole on face\", \n","            \"man with mole near mouth\", \"man with many small moles on face\", \"man with large pimples on face\", \"man with small pimples on face\", \"woman with face scars\", \"woman with wounded face\", \"woman with big mole on face\", \n","            \"woman with mole near mouth\", \"woman with many small moles on face\", \"woman with large pimples on face\", \"woman with small pimples on face\", \"man with congenital moles on face\", \"man with dysplastic nevi on face\", \n","            \"man with acquired nevi on face\", \"man with spitz nevi1 on face\", \"man with whiteheads on face\", \"man with blackheads on face\", \"man with papules on face\", \"man with pustules on face\", \"man with nodules on face\", \n","            \"man with cysts on face\", \"woman with congenital moles on face\", \"woman with dysplastic nevi on face\", \"woman with acquired nevi on face\", \"woman with spitz nevi1 on face\", \"woman with whiteheads on face\", \n","            \"woman with blackheads on face\", \"woman with papules on face\", \"woman with pustules on face\", \"woman with nodules on face\", \"woman with cysts on face\", \"teenage boy with pimples\", \"teenage girl with pimples\", \n","            \"man with eye scar\", \"man with mouth scar\", \"man with lip scar\", \"man with cheek scar\", \"man with forehead scar\", \"man with chin scar\", \"woman with eye scar\", \"woman with mouth scar\", \"woman with lip scar\", \n","            \"woman with cheek scar\", \"woman with forehead scar\", \"woman with chin scar\", \"man with large eyebags\", \"woman with large eyebags\", \"pretty mole on face\", \"model wih mole on face\", \"man with face skin disease\", \n","            \"woman with face skin disease\", \"man with red patches on face\", \"man with purple patches on face\", \"woman with red patches on face\", \"woman with purple patches on face\", \"man with pimple on nose\", \"man with mole on nose\", \n","            \"woman with pimple on nose\", \"woman with mole on nose\", \"man with chickenpox\", \"woman with chickenpox\", \"man with nose scar\", \"woman with nose scar\", \"man with scabby face\", \"woman with scabby face\", \"man with scarry face\", \n","            \"woman with scarry face\", \"man with injured face\", \"woman with injured face\", \"man with freckles on one side of the face\", \"woman wih freckles on one side of face\", \"man with bruises on face\", \"woman with bruises on face\", \n","            \"very big mole on face\", \"scratches on face\"\n","        ],\n","    },\n","    \"neutral\": {\n","        \"\": [\n","            \"black woman\", \"black man\", \"white woman\", \"white man\", \"asian woman\", \"asian man\", \"woman with no makeup\", \"male\", \"female\", \"profile view male\", \"profile view female\", \"mixed race person\", \"person with headwear\", \n","             \"person with eyewear\", \"person with accessories\", \"famous person\", \"young man\", \"old man\", \"young woman\", \"old woman\", \"pretty person\", \"ugly person\", \"person with hat\", \"person with glasses\", \"person with earbuds\", \"person\"\n","        ]\n","    }\n","}\n","\n","NEUTRAL_GROUPS = {\n","    \"no_headwear\": [-18],\n","    \"no_eyewear\": [-6],\n","    \"no_facewear\": [10, 12, 13, -6],\n","    \"no_accessories\": [-9, -15],\n","}"]},{"cell_type":"code","source":["cropper = Cropper(\n","    output_size=512,\n","    output_format=\"jpg\",\n","    face_factor=0.6,\n","    enh_threshold=None,\n","    det_threshold=0.99,\n","    device=\"cuda:0\",\n","    strategy=\"best\",\n","    attr_groups=NEUTRAL_GROUPS if target_dir==\"neutral\" else None,\n",")"],"metadata":{"id":"o2g3wgYj5BIv"},"execution_count":null,"outputs":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5XlemEurrsTf"},"outputs":[],"source":["for group, group_dict in KEYWORD_DICT.items():\n","    if group != target_dir:\n","        continue\n","    \n","    for subgroup, keywords in tqdm(group_dict.items()):\n","        outdir = os.path.join(\"..\", \"images\", group, subgroup)\n","\n","        for keyword in keywords:\n","            download_from_keywords(keyword, outdir)\n","            subdir = os.path.join(outdir, keyword)"]},{"cell_type":"markdown","source":["## Image Processing\n","\n","Here, the images will be processed: faces will be aligned and center-cropped. Note that after they are processed, we need to delete the original images that no longer will be needed. If our target directory is `headwear`, then the original subdirectories are `hoodie`, `helmet` and `top`. The newly created subdirectories with face images (`hoodie_faces`, `helmet_faces` and `top_faces`) can also be renamed without suffix `_faces` before the files are zipped. Once the zip file is created, it can be downloaded if working on _Google Colab_."],"metadata":{"id":"mfRJYUoMbcgz"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"fY6nQMaCkQwx"},"outputs":[],"source":["if target_dir == \"neutral\":\n","    cropper.process_dir(f\"../images/{target_dir}\")\n","else:\n","  for subdir in os.listdir(f\"../images/{target_dir}\"):\n","      print(f\"Processing: {subdir}\")\n","      cropper.process_dir(f\"../images/{target_dir}/{subdir}\")"]},{"cell_type":"code","source":["%cd ../images"],"metadata":{"id":"rQAkK6pKbl2R"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["!rm -r $target_dir/hoodie\n","!rm -r $target_dir/helmet\n","!rm -r $target_dir/top"],"metadata":{"id":"bofweml23yLO"},"execution_count":null,"outputs":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"_jlIl9yaSiiD"},"outputs":[],"source":["!zip -r $target_dir.zip $target_dir"]}],"metadata":{"accelerator":"GPU","colab":{"provenance":[],"authorship_tag":"ABX9TyPVejryuntgAM6cLSSVTAjt"},"gpuClass":"standard","kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}