docker的文件流处理_使用Nifi-Registry和Docker对Nifi流文件进行版本控制和持久化

currently I'm playing with the latest stable version Nifi/Nifi-registry and Docker. I'm trying to version and persist flowfiles in Gitlab.

I found several information in the web but right now it's not working. All components are working and I'm also able to version files from Nifi to Nifi-Registry, but the last step to Gitlab is currently missing.

--> I modified and mounted the providers.xml into the container

org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider

./versioned_flows

origin

*Name*

*Token*

Does andybody have some experiences and maybe a code snippet? I would appreciate that.

Thanks a lot.

Kind regards,

T

解决方案

Although it is an old post, I will be happy if it is helpful to anyone.

I have the following docker folder & it runs both nifi & it's registry containers.

Copy the 'conf' folder inside nifi directory (can be taken from running the container without volume)

Run docker volume create nifi_data

The docker-compose.yml file is:

version: "3.7"

services:

nifi:

container_name: nifi

image: apache/nifi:1.11.4

ports:

- target: 8080

published: 8080

protocol: tcp

mode: host

restart: always

environment:

- NIFI_WEB_HTTP_HOST=0.0.0.0

- NIFI_JVM_HEAP_INIT=4g

- NIFI_JVM_HEAP_MAX=6g

- NIFI_VARIABLE_REGISTRY_PROPERTIES=./conf/env.properties

volumes:

- nifi_data:/opt/nifi/nifi-current/

- ./nifi/extensions:/opt/nifi/nifi-current/extensions

- ./nifi/conf:/opt/nifi/nifi-current/conf

nifi-registry:

container_name: nifi-registry

image: apache/nifi-registry:0.7.0

ports:

- target: 18080

published: 18080

protocol: tcp

mode: host

environment:

- NIFI_REGISTRY_WEB_HTTP_HOST=0.0.0.0

- JVM_OPTS=-Xmx512m -Xms512m -XX:MaxPermSize=1g

volumes:

- $PWD/registry/providers.xml:/opt/nifi-registry/nifi-registry-current/conf/providers.xml

- $PWD/registry/flow-storage:/opt/nifi-registry/nifi-registry-current/flow_storage

- $PWD/registry/database:/opt/nifi-registry/nifi-registry-current/database

volumes:

nifi_data:

external: true

networks:

default:

external:

name: nifi-network

Note: I have custom properties (env.properties) & the custom processors are under 'extensions' directory. Also, you may have permissions issue since nifi runs under 'nifi' user - I am sure you will sort it out :)

The providers.xml (take the default one - comment the filesystem provider & uncomment the git provider)

org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider

./flow_storage

And finally the restart.sh file:

chown -R 1000:1000 nifi registry // nifi's user:group

chmod -R 775 nifi registry

chmod -R g+s nifi registry

docker-compose down

docker-compose up -d