CI

CrowdStrike Falcon Event Stream to Humio

This project intend to provide a simple way of moving your CrowdStrike Falcon Event Stream data into Humio.
As is the only reliable way of getting Event Stream data is with CrowdStrike’s SIEM connector, that dumps to multiple files.
We’re trying to bypass the file stage and ship this directly to Humio and streamline ingest, providing CrowdStrike customers
with Humio a simple way to maintain, visualize and alert on Falcon Event Stream data.

Design

This project is build as a Python package (application) to be shipped within a Docker or other containerized environment.
The application error handling could be better, and the primary way to respond to unexpected errors is to shut down, relying on docker to restart the process.

Prerequisite

For setting up the connection you need two types of credentials ready.

For running the code one of the following is needed.

  • Docker with access to persistent volume
  • Python3 (with virtual environment recommended!)

Obtaining Falcon API key-pair

  • Login to your Falcon Console
  • Go to Menu -> Support -> API Clients and Keys
  • Click Add new API client
    • Set the client a name
    • Optional: fill the description
    • Assign Event streams/read access

falcon_token

Obtaining Humio Ingest Token

  • Login to your Humio cluster

  • Go to the repository you’re going to use

  • Depending on if you’re going to run enriched or not.

    • Enriched: Download the siem-connector-enriched.yaml parser

      • Go to Parsers and click New Parser then select From template

      • Give the parser a name, note, this is going to be assigned the #type field. E.g. siem-connector-enriched

      • Upload the yaml specification. This is for now an empty parser, you can simply create an empty parser yourself as well.

    • Normal: Install the package crowdstrike/siem-connector by doing the following
      Note that enriched event can use thhis content as well

      • Go to Settings -> Marketplace -> crowdstrike/siem-connector and click Install package -> Install

  • Go to Settings -> Ingest tokens and click Add token

    • Give the ingest token a good name

    • Enriched: assign the parser you created in previous step

    • Normal: select the crowdstrike/siem-connector -> siem-connector

humio_token

Installation

We recommend using the docker image unless you plan around creating this as a systemd service or similar.

# Clone the sample environment file
wget https://raw.githubusercontent.com/Trifork-Security/cses2humio/master/cses2humio.env.example -O cses2humio.env

Modify the attributes accordingly, for more information see Command line and arguments

Start the container with the newly configured environment file

docker run -v $HOST_DATA_DIR:/data  \
    --name=cses2humio \
    --env-file=$PATH_TO_CONFIG_FILE \
    --detach --restart=always \
    ghcr.io/trifork-security/cses2humio:latest

See your data coming in!

docker logs -f cses2humio

Command line and arguments

You can specify run arguments as command lines or environment variables (same as command line, just all uppercase)

Argument             Environment Description
–offset-file OFFSET_FILE General: Where to save offsets for partitions. File will be created automatically
Default: offset.db
Note that the cses2humio.env.example defaults to /data/offset.db
–enrich ENRICH General: Parses the events before shipping to Humio, and expands some fields due to such parsing in Humio can be tricky
Default: False
–verbose VERBOSE General: Be verbose, use for debugging and troubleshooting
Default: False
–falcon-url FALCON_URL Falcon: Url to the API, not the console
Default: https://api.crowdstrike.com
–falcon-api-id FALCON_API_ID Falcon: API ID for the created key
Default: N/A
–falcon-api-secret FALCON_API_SECRET Falcon: API Secret for the created key
Default: N/A
–humio-url HUMIO_URL Humio: Url for the Humio Cluster for events to go
Default: https://cloud.humio.com
–humio-token HUMIO_TOKEN Humio: Ingest token, remember to assign correct parser
Default: N/A
–app-id APP_ID Advanced: Specific to Falcon Event Stream, don’t change unless you know what you’re doing!
Default: cses2humio
–user-agent USER_AGENT Advanced: User agent used in HTTP requests
Default: cses2humio/{version}
–bulk-max-size BULK_MAX_SIZE Advanced: Maximum number of events to send in bulk
Default: 200`
–flush-wait-time FLUSH_WAIT_TIME Advanced: Maximum wait time before flushing queue
Default: 10

You can also run the tool directly from commandline (using environment variables as well)

cses2humio -h
usage: cses2humio [-h] [--offset-file OFFSET_FILE] [--enrich] [-v] [--falcon-url FALCON_URL] [--falcon-api-id FALCON_API_ID] [--falcon-api-secret FALCON_API_SECRET] [--humio-url HUMIO_URL] [--humio-token HUMIO_TOKEN] [--app-id APP_ID] [--user-agent USER_AGENT] [--bulk-max-size BULK_MAX_SIZE]
                  [--flush-wait-time FLUSH_WAIT_TIME]

CrowdStrike Falcon Event Stream to Humio

optional arguments:
  -h, --help            show this help message and exit

General:
  --offset-file OFFSET_FILE
                        Location including filename for where to store offsets, default is current directory as offset.db
  --enrich              Will parse some fields as they're hard to parse in Humio.Note this might be more resources intensive but spare Humio of parsing. Default is off
  -v, --verbose         Increase output verbosity

Falcon:
  --falcon-url FALCON_URL
                        Falcon API URL, note this is for the API given when you create the API key. Defaults to US-1 API url
  --falcon-api-id FALCON_API_ID
                        Falcon API ID to use for OAuth2
  --falcon-api-secret FALCON_API_SECRET
                        Falcon API Secret to use for OAuth2

Humio:
  --humio-url HUMIO_URL
                        Humio URL for the cluster going to ingest data. Default to https://cloud.humio.com
  --humio-token HUMIO_TOKEN
                        Ingest token to use for ingesting data. Remember to assign the correct parser depending on parsing

Advanced:
  --app-id APP_ID       App ID to use for consuming events
  --user-agent USER_AGENT
                        User agent used to connect to services
  --bulk-max-size BULK_MAX_SIZE
                        Maximum number of events to send in bulk
  --flush-wait-time FLUSH_WAIT_TIME
                        Maximum time to wait if bulk max size isn't reached

Building

# Clone the repo and switch to it
git clone https://github.com/Trifork-Security/cses2humio.git
cd cses2humio

# Create virtual environment and activate (optional, but recommended)
python3 -m venv venv
source venv/bin/activate

# Install build and build the package (used in Dockerfile)
pip3 install build
python3 -m build 

# Build the docker image
docker build -t [TAG_FOR_IMAGE] .

Contributing

Please feel free to contribute at any time by doing a PR.

License

Apache License 2.0

GitHub

View Github