Spacy Task

We have also included a simply Spacy Disdat task. All it does is load up a Spacy model. Here we just want to show how to build your container to contain some non-Python code.

The $CODE/pipelines/en_core_web_sm directory contains the English-trained CNN. We need to have this data in our container. We do that by:

• Create a MANIFEST.in file at the same level as setup.py

• And inside our setup.py we have the include_package_data=True directive.

$more $CODE/MANIFEST.in
recursive-include pipelines/en_core_web_sm *

You've already built the container in the previous step with the MNIST example. So you can simply run the simple pipeline to see that it works.

$dsdt run . pipelines.nlp_spacy.SimpleNLP
...
===== Luigi Execution Summary =====

Scheduled 2 tasks of which:
* 2 ran successfully:
    - 1 DriverTask(...)
    - 1 SimpleNLP(...)

This progress looks :) because there were no failed tasks or missing dependencies