tap-gitlab

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from GitLab's REST API
• Extracts the following resources from GitLab:
  • Branches
  • Commits
  • Issues
  • Projects
  • Project milestones
  • Users
  • Groups
  • Group Milestones
• Outputs the schema for each resource
• Incrementally pulls data based on the input state

Quick start

1. Install

   > pip install tap-gitlab

2. Get your GitLab access token

   • Login to your GitLab account
   • Navigate to your profile page
   • Create an access token

3. Create the config file

   Create a JSON file called config.json containing:

   • Access token you just created
   • (Optional) api_url — the base URL of your GitLab instance, without the /api/v4 path (e.g. https://gitlab.mycompany.com). Defaults to https://gitlab.com. Set this only when connecting to a self-hosted / on-premises GitLab server. The tap appends /api/v4 automatically.
   • Groups to track (space separated)
   • Projects to track (space separated)

   Notes:

   • either groups or projects must be provided.
   • groups only — the tap fetches all projects belonging to those groups and syncs them along with their child streams (branches, commits, issues, etc.).
   • projects only — the tap syncs the explicitly listed projects and their child streams.
   • groups and projects both — the tap merges projects from both sources (deduplicating any overlap) and syncs all of them. The projects stream independently queries the groups API; it does not depend on the groups stream being selected.
   • The groups stream and its child group_milestones are independent of the projects tree and can be selected or deselected without affecting project data sync.
   • groups accepts a space or comma-separated list of group IDs.
   • projects accepts a space or comma-separated list of project IDs.

   GitLab Cloud (default):

   {
       "private_token": "your-access-token",
       "groups": "myorg mygroup",
       "projects": "project_id1 project_id2",
       "start_date": "2018-01-01T00:00:00Z"
   }

   Self-hosted / on-premises GitLab:

   {
       "private_token": "your-access-token",
       "api_url": "https://gitlab.mycompany.com",
       "groups": "myorg mygroup",
       "projects": "myorg/repo-a myorg/repo-b",
       "start_date": "2018-01-01T00:00:00Z"
   }

4. [Optional] Create the initial state file

   You can provide JSON file that contains a date for the API endpoints to force the application to only fetch data newer than those dates. If you omit the file it will fetch all GitLab data

   {
       "branches": "2017-01-17T00:00:00Z",
       "commits": "2017-01-17T00:00:00Z",
       "issues": "2017-01-17T00:00:00Z",
       "projects": "2017-01-17T00:00:00Z",
       "users": "2017-01-17T00:00:00Z",
       "group_milestones": "2017-01-17T00:00:00Z"
   }

   Note:

   • currently, groups don't have a date field which can be tracked

5. Run the application

   tap-gitlab can be run with:

   For Sync mode:

   > tap-gitlab --config tap_config.json --catalog catalog.json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To load to json files to verify outputs:

   > tap-gitlab --config tap_config.json --catalog catalog.json | target-json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To pseudo-load to Stitch Import API with dry run:

   > tap-gitlab --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

---

Copyright © 2018 Stitch