Getting Started With The Njit Highlander Pipeline
I first ran into this when a professor asked our lab group to process a batch of sensor data from a campus environmental monitoring project. The pipeline was supposed to handle ingestion, validation, transformation, and export. What followed was roughly three weeks of debugging because nobody had documented the intermediate schema changes between the v2 and v3 release. The core idea is straightforward enough, but the execution has enough moving parts that you will trip over something if you assume the examples in the README will work out of the box. The pipeline is an open-source ETL framework designed for academic research and small-scale production workflows at NJIT. It handles data ingestion from multiple sources, applies configurable transformation rules, and outputs structured datasets that downstream tools can consume. It is not a general-purpose solution. It works well if your data volume stays under a few hundred gigabytes and your transformations are rule-based rather than model-driven. Beyond that, you will start hitting resource constraints that make the setup feel heavier than it needs to be. Here is how to actually get it running on your machine without the usual frustration.
Installation And Setup
Clone the repository from the official NJIT GitHub org. The repo is under the highlander-pipeline namespace. Make sure you are using Python 3.10 or later. The project pins numpy to a specific range, and newer minor releases of Python break the build in ways that are not obvious from the error output. After cloning, create a virtual environment and install the requirements. I recommend running pip install -e . so you can modify source files without reinstalling. The package includes a configuration file at config/pipeline.yaml. Do not skip reading through this file before you run anything. It controls the input paths, transformation modules, and output destinations. Once the config is in place, you run the pipeline with:
python -m highlander_pipeline run --config config/pipeline.yaml This executes the default pipeline stages in order. You will see progress logs written to stdout and a JSON status file in the output/ directory when the run completes. The status file tells you whether each stage passed, failed, or was skipped. That file is your first diagnostic tool when something goes wrong.
Get the Full Details
Configuring Your First Run
The configuration file uses YAML. The structure has three main sections: inputs, transforms, and outputs. The inputs section defines where data comes from. You can specify local file paths, CSV directories, or API endpoints. For local files, the pipeline supports glob patterns, which is useful when you have date-partitioned data. The transforms section is where most people spend their time. Each transform is a named module with configurable parameters. Common transforms include schema validation, type coercion, null handling, and custom Python functions. The custom function transform lets you inject arbitrary Python code that operates on each row or batch. This is powerful but dangerous if you do not validate your code against a small sample first. The outputs section defines where processed data goes. Supported formats include Parquet, CSV, and JSON Lines. Parquet is the default recommendation because it preserves column types and compresses well. If you output to CSV, you will lose type information and row counts can balloon quickly.
A Real Problem I Encountered
During a project last semester, the pipeline was failing silently on the validation stage. No error message, no status update, just a skipped stage. I spent about four hours checking the transform modules before I realized the issue was with the input file encoding. One of the source CSV files was UTF-8 with BOM, and the reader was treating the BOM as part of the first column name. That caused a schema mismatch that the validator caught but did not report properly. The workaround was straightforward once I found it. I added a preprocessing step that strips the BOM before the data hits the validation stage. In the config, I inserted a new transform at the top of the transforms list: { "name": "strip_bom", "module": "highlander.transforms.preprocessing", "params": { "encoding": "utf-8-sig" } }
That fixed it. Going forward, I always run a file scan on my input data before configuring the pipeline. A quick command like file -i * on Unix or checking encoding with a small Python script saves a lot of debugging time.
Common Pitfalls And Advanced Nuances
Most beginners treat the pipeline as a linear process. It is not. The framework supports branching and conditional execution, but that functionality is not well documented. If you need to route data differently based on its content, you can use the conditional transform, but you have to define the conditions in the config file using a simple expression language. It works, but the syntax is easy to get wrong, and errors in conditions result in silently skipped branches rather than failures, which makes debugging confusing. Another thing that trips people up is the difference between row-level and batch-level transforms. Row-level transforms process one record at a time, which is memory efficient but slow for large datasets. Batch-level transforms process chunks, which is faster but can cause memory issues if your batches are too large. The default batch size is 10,000 rows. I usually set it to 50,000 for datasets under 5 GB and drop it to 5,000 when I am working with limited RAM. The pipeline also does not handle incremental updates well. If you rerun the pipeline on the same input directory, it processes everything from scratch. There is no built-in change data capture or delta detection. For recurring jobs, you need to implement your own partitioning strategy or use a tool like Apache Airflow alongside the pipeline to track what has already been processed.
Performance Expectations
On a standard laptop with 16 GB of RAM, the pipeline processes roughly 500 MB of CSV data per minute through a typical three-transform workflow. Parquet output is significantly faster to write than CSV because the pipeline can stream directly without type conversion overhead. If your workflow involves custom Python transforms, expect a 3x to 5x slowdown compared to built-in transforms. This is expected behavior since custom transforms run in Python rather than through optimized C libraries. For larger datasets, the pipeline supports distributed execution through Dask, but that requires additional setup and a cluster environment. The single-node mode is where most students and researchers stay, and it is perfectly adequate for research-grade data processing.
Where The Pipeline Falls Short
The documentation is incomplete. Several transform modules exist in the codebase but have no README entry. The error messages are sometimes unhelpful, particularly when a transform fails due to a missing dependency rather than a configuration issue. The project has sporadic update cycles, so features may appear and then sit undocumented for months. If you need enterprise-grade monitoring, alerting, or SLA guarantees, this is not the right tool. For academic projects, capstone work, or small research datasets, it is functional and free. If your needs grow beyond that, look at Apache Spark or Prefect as alternatives. They have steeper learning curves but better long-term support.
Getting The Code
The repository is available on GitHub under the NJIT organization. Search for highlander-pipeline to find it. The README includes a quickstart guide and a list of available transforms. Beyond that, the source code is the best documentation you will find for edge cases. The transform modules are well structured and easy to read if you need to understand what is happening under the hood.
