Getting Started With NDCJ: A Practical Walkthrough
I still remember the first time I tried to work with NDCJ, sitting at my desk at 11pm because the production deadline wasn't going to meet itself. The documentation was decent but fragmented across a few outdated wiki pages and a GitHub repo that hadn't been updated in eight months. What follows is the actual process, not the polished version. NDCJ stands for the National Digital Conversion and Journalism initiative, though in practice people tend to just call it the conversion pipeline. It is a framework designed to standardize how newsroom content gets digitized, metadata-tagged, and distributed across multiple platforms simultaneously. The "art" part is the editorial judgment that goes into tagging and structuring. The "science" part is the automated pipeline that handles format conversion, compression, and cross-platform optimization. Here is what most people miss when they start: NDCJ is not a single piece of software. It is a stack. You have the ingest layer, the processing layer, and the delivery layer, and each one can run independently if your operation is small enough. When I first set this up for a regional news desk, we only needed the ingest and delivery layers. The processing layer added about forty-five minutes of turnaround per article and honestly wasn't worth it for our volume. We disabled it and wrote a custom Python script that handled our specific metadata requirements in about twelve minutes per piece.
Installation and Initial Setup
You will need Docker and Docker Compose running. Pull the latest image from the official registry — the tag is ndcj/pipeline:stable. There is a legacy tag called ndcj/pipeline:latest but do not use it. That branch has a known memory leak that causes the worker process to crash after roughly six hours of continuous operation. I learned that the hard way on a Sunday night during a breaking news cycle. Create a docker-compose.yml file with the following structure: services:
ndcj-ingest:
image: ndcj/pipeline:stable
ports:
- "8080:8080"
volumes:
- ./ingest-data:/data/ingest
- ./config:/app/config
environment:
- ND CJ_ENV=production
- ND CJ_STORAGE_BACKEND=s3
- AWS_BUCKET_NAME=your-bucket-name
After the containers come up, navigate to localhost:8080/admin and create your first user account. The default credentials are documented but you should change them immediately. I have seen three separate newsrooms compromise their pipelines in a single quarter because someone left the default admin password in place.
Get the Full Details
Configuring the Pipeline
The configuration lives in config/pipeline.yaml inside your mounted volume. The critical section is the processing_rules block. Here is where you define how assets move through the system. A typical rule set looks like this: processing_rules:
image_optimization:
enabled: true
format: webp
quality: 82
max_width: 1920
metadata_extraction:
enabled: true
plugins:
- exif_parser
- article_structure_analyzer
- sentiment_tagger
delivery_targets:
- type: web
priority: high
- type: amp
priority: medium
- type: social_feed
priority: low The quality setting of 82 is important. Going lower than 78 introduces visible artifacts on photographs that contain text — like signs, documents, or screens. Going higher than 85 wastes bandwidth with negligible visual improvement on modern displays. I ran a blind test once with twenty editors comparing 82 versus 90 and nobody could tell the difference on 100 out of 120 images. The remaining 20 were close-ups of text-heavy content, which is why the max_width cap matters as much as the quality setting.
Common Pitfalls and Workarounds
The biggest issue I encounter in production environments is the exif_parser plugin choking on raw camera files. It works fine with JPEGs and standard PNGs, but when someone uploads a .CR3 or .NEF file, the pipeline either errors out or takes up to four minutes to process a single image. The workaround is to add a pre-processing step that converts raw files to JPEG before they hit the main pipeline. I wrote a simple ffmpeg-based converter that runs as a separate Docker container and sits between the ingest endpoint and the main pipeline. It reduces raw file handling time from around three minutes to about eight seconds. Another problem is the sentiment_tagger plugin producing inconsistent results when processing non-English content. The model was trained primarily on American English news corpus, so articles in other languages or featuring heavy regional dialects get tagged incorrectly roughly thirty percent of the time. If your newsroom covers multilingual content, disable that plugin and use a language-specific tagging pipeline instead. The NDCJ architecture supports chained plugins, so you can route different language streams to different processors without conflict. There is also a rate limiting issue that catches people off guard. The default configuration allows 200 requests per minute to the delivery API. If you are pushing content during a breaking story — say, twelve articles with an average of six images each over a ten-minute window — you will hit that limit and the excess requests queue up with no clear notification. The pipeline does not throw an error. It just silently delays delivery. I added a monitoring script that checks the queue depth every thirty seconds and sends an alert when it exceeds fifty items. This caught the issue for me twice in the first month before I realized it was a rate limit problem rather than a system failure.
Performance Expectations
Under normal conditions, a single article with five images and two video clips processes through the full pipeline in about three to five minutes on a modest setup — two CPU cores, eight gigabytes of RAM, and a solid state drive for local caching. If you are running this on cloud infrastructure with auto-scaling enabled, that drops to under two minutes. The bottleneck is almost always the image optimization step, specifically when the quality setting is above 85 or when the source images are unusually large. A typical feature story with high-resolution photography can push the processing time to eight or nine minutes because the pipeline is resizing and recompressing forty or fifty individual images sequentially. Scaling beyond a single node is straightforward but not trivial. You add workers by increasing the replica count in your docker-compose file, but you need to ensure the storage backend is shared across all nodes. S3 works well for this. Local disk mounts do not. I watched a team try to run four worker nodes with local storage and end up with articles being delivered in completely different formats depending on which worker happened to pick up the job. The inconsistency was subtle — one version had slightly different color profiles, another had missing alt text metadata — but it became a real problem when the same article went live on multiple platforms simultaneously. The system does have limitations. It cannot handle video transcoding beyond H.264 at reasonable scale. If you need H.265 or VP9 output, you will need to integrate a separate transcoding service like FFmpeg on its own and point the delivery layer to it. The documentation mentions this briefly in the advanced integrations section but does not give you a working example. I spent about a day getting the integration working and wrote up the configuration, which you can find on the community GitHub under the video-integration examples folder. It is not officially supported, but it works.
Another hard limit is around concurrent user access. The admin panel and API support roughly thirty simultaneous authenticated users before response times start degrading noticeably. If your newsroom is larger than that, you need to implement role-based access control and restrict who has API write access. Most teams only need five to ten people writing to the pipeline directly. The rest should be reading from it or consuming the delivered content through the CMS integration.
Download and Resources
The pipeline is available at the official NDCJ repository on GitHub. The README there is adequate but does not cover the edge cases I mentioned. For the raw installation package and all plugin sources, you can download directly from the releases page. The stable branch is tagged v2.4.1 as of this writing. Avoid the development branches unless you are comfortable debugging untested code in a production environment. The community Discord server is active enough that you can get answers to basic setup questions within a few hours. The maintainers respond to serious issues but tend to be slow on feature requests. If you need something the pipeline does not currently support, you are more likely to get a working solution by building it yourself using the plugin SDK than by waiting for an upstream update. There is also a paid support tier if your organization needs guaranteed response times or custom development work. The pricing is reasonable for what you get, but honestly most problems that trigger the need for paid support could have been solved with enough reading of the source code and trial-and-error. I only used paid support once, and it was because a misconfigured S3 bucket policy was causing silent data loss — the kind of issue that is hard to debug without someone who has seen it before.
