Create Gitlab Personal Access Token using curl

gitlab_access_token.md

Create Gitlab Personal Access Token using curl

Prerequisites

• You need a Gitlab server up and running
• You need user credentials for a (admin) user on the Gitlab server
• You need curl and Perl on your server

What does it do?

1. Open the login page of Gitlab to get session cookie
2. Login to Gitlab using username and password to get authenticated session cookie
3. Open the Personal Access Tokens page
4. POSTing the Personal Access Token for to generate a personal access token
5. Scrape the personal access token from the returned HTML page

Code

gitlab_host="http://localhost:8080"
gitlab_user="root"
gitlab_password="12341234"

# 1. curl for the login page to get a session cookie and the sources with the auth tokens
body_header=$(curl -c cookies.txt -i "${gitlab_host}/users/sign_in" -s)

# grep the auth token for the user login for
#   not sure whether another token on the page will work, too - there are 3 of them
csrf_token=$(echo $body_header | perl -ne 'print "$1\n" if /new_user.*?authenticity_token"[[:blank:]]value="(.+?)"/' | sed -n 1p)

# 2. send login credentials with curl, using cookies and token from previous request
curl -b cookies.txt -c cookies.txt -i "${gitlab_host}/users/sign_in" \
	--data "user[login]=${gitlab_user}&user[password]=${gitlab_password}" \
	--data-urlencode "authenticity_token=${csrf_token}"

# 3. send curl GET request to personal access token page to get auth token
body_header=$(curl -H 'user-agent: curl' -b cookies.txt -i "${gitlab_host}/profile/personal_access_tokens" -s)
csrf_token=$(echo $body_header | perl -ne 'print "$1\n" if /authenticity_token"[[:blank:]]value="(.+?)"/' | sed -n 1p)

# 4. curl POST request to send the "generate personal access token form"
#      the response will be a redirect, so we have to follow using `-L`
body_header=$(curl -L -b cookies.txt "${gitlab_host}/profile/personal_access_tokens" \
	--data-urlencode "authenticity_token=${csrf_token}" \
	--data 'personal_access_token[name]=golab-generated&personal_access_token[expires_at]=&personal_access_token[scopes][]=api')

# 5. Scrape the personal access token from the response HTML
personal_access_token=$(echo $body_header | perl -ne 'print "$1\n" if /created-personal-access-token"[[:blank:]]value="(.+?)"/' | sed -n 1p)

Usage of token in API Requests

According to the Gitlab API documentation, you can now use the personal_access_token to make API requests:

curl --header "Private-Token: ${personal_access_token}" https://gitlab.example.com/api/v4/projects

Hmm I think this doesn't work anymore, because of possibly one of two things. 1) Since GitLab does that thing where it checks your browser before letting you log in, or 2) it doesn't work because I have MFA set up on my GitLab account 😢

@michaellihs Does this script account for any of that?

@emmahsax Same here with an on prem install (13.8.x) without MFA. My guess is that something has been changed in Gitlab that this doesn't work anymore. The script just returns empty output

Came across the same issue for Gitlab IaC...got around it by using rails console to set an arbitrary token.

https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#programmatically-creating-a-personal-access-token

gitlab-rails runner can be replaced with bundle exec rails runner -e production if needed

Really appreciate your help.
Just for data integrity want to leave link to SO with the same question:
https://stackoverflow.com/questions/47948887/login-to-gitlab-with-username-and-password-using-curl

And a few notes from myself:

1. It did not work for 'as is';
2. added -L redirect parameter to all curl requests (got 302 before that);
3. changed url because as checked with UI there was extra -/ in path. So finally in script I use:

"${gitlab_host}/-/profile/personal_access_tokens"

4. Got some 422 errors, maybe my VM connection issue. At the 2-3rd try the TOKEN was created, received in script, confirmed in UI.

---

Self-hosted Gitlab as docker image gitlab/gitlab-ee:latest:
sha256:fb63c8733903dfffd659bdb31efbf586c9fb7bde127136c29d55ac2f26f4f2ee
version: 14.8.2-ee.0
my image page: https://hub.docker.com/layers/gitlab/gitlab-ee/latest/images/sha256-fb63c8733903dfffd659bdb31efbf586c9fb7bde127136c29d55ac2f26f4f2ee?context=explore

still useful, thanks man.

gitlab_host="http://localhost:8080"
gitlab_user="root"
gitlab_password="12341234"

# 1. curl for the login page to get a session cookie and the sources with the auth tokens
body_header=$(curl -c cookies.txt -i "${gitlab_host}/users/sign_in" -s)
# grep the auth token for the user login for
#   not sure whether another token on the page will work, too - there are 3 of them
csrf_token=$(echo $body_header | perl -ne 'print "$1\n" if /new_user.*?authenticity_token"[[:blank:]]value="(.+?)"/' | sed -n 1p)

# 2. send login credentials with curl, using cookies and token from previous request
curl -L -b cookies.txt -c cookies.txt -i "${gitlab_host}/users/sign_in" \
  --data-raw "user%5Blogin%5D=${gitlab_user}&user%5Bpassword%5D=${gitlab_password}" \
  --data-urlencode "authenticity_token=${csrf_token}" \
  --compressed \
  --insecure 2>&1 > /dev/null

# 3. send curl GET request to personal access token page to get auth token
body_header=$(curl -H 'user-agent: curl' -b cookies.txt -i "${gitlab_host}/-/profile/personal_access_tokens" -s)
csrf_token=$(echo $body_header | perl -ne 'print "$1\n" if /authenticity_token"[[:blank:]]value="(.+?)"/' | sed -n 1p)

# 4. curl POST request to send the "generate personal access token form"
#      the response will be a redirect, so we have to follow using `-L`
body_header=$(curl -L -b cookies.txt "${gitlab_host}/-/profile/personal_access_tokens" \
    --data-urlencode "authenticity_token=${csrf_token}" \
    --data 'personal_access_token[name]=golab-generated&personal_access_token[expires_at]=&personal_access_token[scopes][]=api')

# 5. Scrape the personal access token from the response json
personal_access_token=$(echo $body_header | perl -ne 'print "$1\n" if /new_token":"(.+?)"/' | sed -n 1p)

version GitLab Community Edition [15.4.2]

gitlab_host="https://"
gitlab_user="root"
gitlab_password=""

# 1. curl for the login page to get a session cookie and the sources with the auth tokens
body_header=$(curl -c cookies.txt -i "${gitlab_host}/users/sign_in" -s --insecure)
# grep the auth token for the user login for
#   not sure whether another token on the page will work, too - there are 3 of them
csrf_token=$(echo $body_header | perl -ne 'print "$1\n" if /new_user.*?authenticity_token"[[:blank:]]value="(.+?)"/' | sed -n 1p)
# 2. send login credentials with curl, using cookies and token from previous request
curl -L -b cookies.txt -c cookies.txt -i "${gitlab_host}/users/sign_in" \
  --data-raw "user%5Blogin%5D=${gitlab_user}&user%5Bpassword%5D=${gitlab_password}" \
  --data-urlencode "authenticity_token=${csrf_token}" \
  --compressed \
  --insecure 2>&1 > /dev/null

# 3. send curl GET request to personal access token page to get auth token
body_header=$(curl -H 'user-agent: curl' -b cookies.txt -i "${gitlab_host}/-/profile/personal_access_tokens" -s --insecure)

csrf_token=$(echo $body_header | perl -ne 'print "$1\n" if /csrf-token"[[:blank:]]content="(.+?)"/' | sed -n 1p)
# 4. curl POST request to send the "generate personal access token form"
#      the response will be a redirect, so we have to follow using `-L`
body_header=$(curl -L -b cookies.txt "${gitlab_host}/-/profile/personal_access_tokens" \
    --data-urlencode "authenticity_token=${csrf_token}" \
    --data 'personal_access_token[name]=golab-generated&personal_access_token[expires_at]=&personal_access_token[scopes][]=api' --insecure)
# 5. Scrape the personal access token from the response json
personal_access_token=$(echo $body_header | perl -ne 'print "$1\n" if /new_token":"(.+?)"/' | sed -n 1p)
echo $personal_access_token

version GitLab Community Edition [15.6]

Thanks for your work but do have any clue how to make it work with LDAP authentication ? Thanks
It is also interesting - is there an opportunity to adapt this method for programmatic LDAP authorization? I want to request pages with changes from the Github repository, but I can't go beyond authorization...

Does anyone tried the python module "python-gitlab" or cmdline tool glab, maybe it's a better choice ?

gitlab_host="https://"
gitlab_user=""
gitlab_password=""

# 1. curl for the login page to get a session cookie and the sources with the auth tokens
body_header=$(curl -k -c cookies.txt -i "${gitlab_host}/users/sign_in" -s)

# grep the auth token for the user login for
#   not sure whether another token on the page will work, too - there are 3 of them
csrf_token=$(echo $body_header | perl -ne 'print "$1\n" if /gl-show-field-errors.*?authenticity_token"[[:blank:]]value="(.+?)"/' | sed -n 1p)

# 2. send login credentials with curl, using cookies and token from previous request
curl -k -b cookies.txt -c cookies.txt -i "${gitlab_host}/users/sign_in" \
	--data "user[login]=${gitlab_user}&user[password]=${gitlab_password}" \
	--data-urlencode "authenticity_token=${csrf_token}"

# 3. send curl GET request to personal access token page to get auth token
body_header=$(curl -k -H 'user-agent: curl' -b cookies.txt -i "${gitlab_host}-/user_settings/personal_access_tokens" -s)
csrf_token=$(echo $body_header | perl -ne 'print "$1\n" if /csrf-token"[[:blank:]]value="(.+?)"/' | sed -n 1p)

# 4. curl POST request to send the "generate personal access token form"
#      the response will be a redirect, so we have to follow using `-L`
body_header=$(curl -k -L -b cookies.txt "${gitlab_host}-/user_settings/personal_access_tokens" \
	--data-urlencode "authenticity_token=${csrf_token}" \
	--data 'personal_access_token[name]=golab-generated&personal_access_token[expires_at]=&personal_access_token[scopes][]=api')

# 5. Scrape the personal access token from the response HTML
personal_access_token=$(echo $body_header | perl -ne 'print "$1\n" if /new_token"[[:blank:]]value="(.+?)"/' | sed -n 1p)

version GitLab Community Edition [16.11]

I wanted similar thing (i.e automation of PAT token generation) but for github enterprise server (aka on-prem github) .

Referring this script , I was able to login to our github enterprise server using username, password , setting cookies,header & this authenticity_token etc .
I was also able to generate PAT token of logged in user by firing POST curl -L(L for redirect) on /settings/tokens endpoint , however I am not able to derive its actual value (the one of format ghp_.........) , as HTML response of curl -L /settings/tokens endpoint doesn't contain newly generated PAT.
PS : New PAT is actually getting created , as I do see its name when refreshing https://<my_server>/settings/tokens in browser. Verified by seeing newly generated token's name [not the value , as value is one time watch, you can't see afterwards]. So yeah token was generated through above curl call

To debug this further (i.e why HTML response of curl -L /settings/tokens endpoint doesn't contain newly generated PAT) , Executed this flow in browser manually & observed request calls in dev tools. Noticed here that , when PAT gets generated , its actual value (i.e ghp_.....) gets displayed on page (for one time) & checking HTML response of /settings/tokens endpoint , I do see it contains generated PAT token's value. Also with these I also do see some js/css calls being fired in network request pane simultaneously before new PAT was generated.
So It seems github is doing something extra through css/js to feed this information ( in HTML response (of /settings/tokens endpoint) later & this makes automation of PAT token generation for github difficult (until we use some broswer mocking tools like selenium)

gitlab_host="https://"
gitlab_user=""
gitlab_password=""

# 1. curl for the login page to get a session cookie and the sources with the auth tokens
body_header=$(curl -k -c cookies.txt -i "${gitlab_host}/users/sign_in" -s)

# grep the auth token for the user login for
#   not sure whether another token on the page will work, too - there are 3 of them
csrf_token=$(echo "$body_header" | grep -o 'name="authenticity_token" value="[^"]*"' | head -n1 | sed 's/.*value="\([^"]*\)".*/\1/')
echo "$csrf_token"

# 2. send login credentials with curl, using cookies and token from previous request
curl -k -b cookies.txt -c cookies.txt -i "${gitlab_host}/users/sign_in" \
	--data-urlencode "user[login]=${gitlab_user}" \
	--data-urlencode "user[password]=${gitlab_password}" \
	--data-urlencode "authenticity_token=${csrf_token}"

# 3. send curl GET request to personal access token page to get auth token
body_header=$(curl -k -H 'user-agent: curl' -b cookies.txt -i "${gitlab_host}/-/user_settings/personal_access_tokens" -s)
echo "$body_header"
csrf_token=$(echo "$body_header" | grep -o 'name="csrf-token" content="[^"]*"' | head -n1 | sed 's/.*content="\([^"]*\)".*/\1/')
echo "$csrf_token"

# 4. curl POST request to send the "generate personal access token form"
#      the response will be a redirect, so we have to follow using `-L`
body_header=$(curl -k -L -b cookies.txt "${gitlab_host}/-/user_settings/personal_access_tokens" \
    --request POST \
    --header "Content-Type: application/x-www-form-urlencoded" \
    --header "X-CSRF-Token: ${csrf_token}" \
    --data-urlencode "authenticity_token=${csrf_token}" \
    --data-urlencode "personal_access_token[name]=golab-generated" \
    --data-urlencode "personal_access_token[expires_at]=" \
    --data-urlencode "personal_access_token[scopes][]=api")

# 5. Scrape the personal access token from the response HTML
personal_access_token=$(echo "$body_header" | grep -o '"new_token":"[^"]*"' | sed 's/"new_token":"\([^"]*\)"/\1/')

version GitLab Community Edition [17.5]