Greetings-
Thanks so much for the support of my random questions, its been helpful so far. While deploying Nuxeo in AWS, we're exploring integrating s3 as a blob provider store for Nuxeo.
I'm inquiring on how to get nuxeo to store my asset in s3, right now it seems to be putting it in MongoDB's NoSQL store. Is this desired behavior? Is it only possible to do one or the other (s3 or mongoDB store?) or is it to do both (store in both s3 AND mongoDB?).
Do I need to mount the bucket with something like s3fuse to the OS to a directory path that is defined in nuxeo.conf for repository.binary.store?
I've been reading the source here: https://github.com/nuxeo/cloudbinarymanager/blob/master/src/main/java/org/nuxeo/ecm/core/storage/sql/S3BinaryManager.java and it looks like it interacts with the AWS API to use the bucket, so I can only assume mounting it with some external fuse mount isn't needed.
Any direction is appreciated here.
current setup:
- Nuxeo-Platform: v8.10 (6 nodes 3 AZ's)
- Amazon-s3-Online-Storage Plugin(manually installed): 1.7.3
- MongoDB: v3.2 ( 3 nodes, 1 Primary, 2 secondaries, 3 AZ)
- RDS PostgreSQL: v9.5.4 (2 nodes, 1 Primary, 1 secondary, 2 AZ)
- ElasticCache Redis: v2.8.23 (3 nodes, 3 AZ)
- ElasticSearch: v2.3.5 (13 nodes, 3 Master, 3 Ingest, 3 Client, 4 Data, 3AZ)
nuxeo.conf contents:
##-----------------------------------------------------------------------------
## S3 Bucket settings
##-----------------------------------------------------------------------------
# set default binary manager
nuxeo.core.binarymanager=org.nuxeo.ecm.core.storage.sql.S3BinaryManager
# s3 endpoint
nuxeo.s3storage.endpoint=https://s3-us-west-2.amazonaws.com
# s3 bucket name
nuxeo.s3storage.bucket=some-valid-bucket-name
# s3 region
nuxeo.s3storage.region=us-west-2
# s3 sub directory name (Tier)
nuxeo.s3storage.bucket_prefix=nuxeo/ci/
# Files retrieved from S3 are cached locally for speed.
# bytes or with the standard KB, MB, GB or TB suffixes,
nuxeo.s3storage.cachesize=100MB
# maximum number of files in the cache
nuxeo.s3storage.cachecount=10000
# minimum age (in seconds) a file should have before being eligible for purge (the age is the time since last file access).
nuxeo.s3storage.cacheminage=3600
# configure downloads to be directly served to the user from S3 without going through Nuxeo
# nuxeo.s3storage.directdownload=true
# expire time is expressed in seconds (the default is one hour) and determines how long the generated S3 URLs are valid.
# Having short-lived URLs is better for security, but too short an expiration time could be problematic
# nuxeo.s3storage.directdownload.expire=3600
# internal S3 connection pool settings
# default 50 max connections
nuxeo.s3storage.connection.max=50
nuxeo.s3storage.connection.retry=3
nuxeo.s3storage.connection.timeout=50000
nuxeo.s3storage.socket.timeout=50000
##-----------------------------------------------------------------------------
## Clustering settings
##-----------------------------------------------------------------------------
repository.clustering.enabled=true
repository.clustering.id=167773811
repository.clustering.delay=1
repository.clustering.invalidation=redis
nuxeo.db.validationQuery=SELECT 1
### If clustering is activated, set
repository.binary.store=/opt/nuxeo/binaries
/opt/nuxeo/current/nxserver/config/default-repository-config.xml:
<component name="default-repository-config">
<extension target="org.nuxeo.ecm.core.blob.BlobManager" point="configuration">
<blobprovider name="default">
<class>org.nuxeo.ecm.core.storage.sql.S3BinaryManager</class>
<property name="path">/opt/nuxeo/binaries</property>
<property name="key"></property>
</blobprovider>
</extension>
<extension target="org.nuxeo.ecm.core.storage.mongodb.MongoDBRepositoryService"
point="repository">
<repository name="default" label="label.default.repository">
<server>mongodb://primarydb01.somecompanyurl.internal,secondarydb01.somecompanyurl.internal,secondarydb02.somecompanyurl.internal/?maxPoolSize=200</server>
<dbname>nuxeo</dbname>
<fulltext disabled="false" searchDisabled="false" />
<cache enabled="true" maxSize="1000" concurrencyLevel="10" ttl="10" />
<clustering id="167773811" enabled="true">
<invalidatorClass>org.nuxeo.ecm.core.redis.contribs.RedisDBSClusterInvalidator</invalidatorClass>
</clustering>
</repository>
</extension>
</component>
server.log entries after importing an asset, note the ID.
2017-04-04 15:34:56,240 INFO [Nuxeo-Work-pictureViewsGeneration-1:default:e55408da-a06b-48aa-814a-ab36395ed16d:file:content:pictureView] [org.nuxeo.ecm.automation.core.impl.OperationServiceImpl]
2017-04-03 15:34:56,444 INFO [Nuxeo-Work-pictureViewsGeneration-1:default:e55408da-a06b-48aa-814a-ab36395ed16d:file:content:pictureView] [org.nuxeo.ecm.automation.core.impl.OperationServiceImpl]
2017-04-03 15:34:56,575 INFO [Nuxeo-Work-pictureViewsGeneration-1:default:e55408da-a06b-48aa-814a-ab36395ed16d:file:content:pictureView] [org.nuxeo.ecm.automation.core.impl.OperationServiceImpl]
2017-04-03 15:34:56,595 INFO [Nuxeo-Work-pictureViewsGeneration-1:default:e55408da-a06b-48aa-814a-ab36395ed16d:file:content:pictureView] [org.nuxeo.ecm.automation.core.impl.OperationServiceImpl]
2017-04-03 15:34:56,618 INFO [Nuxeo-Work-pictureViewsGeneration-1:default:e55408da-a06b-48aa-814a-ab36395ed16d:file:content:pictureView] [org.nuxeo.ecm.automation.core.impl.OperationServiceImpl]
My selecting the ID from the mongoDB to find the asset.
mongos> db.default.find({ "ecm:fulltextJobId" :"e55408da-a06b-48aa-814a-ab36395ed16d" })
{ "_id" : ObjectId("58deaf60eb4c4b0b6a4f5ad2"), "ecm:racl" : [ "Administrator", "members" ], "icon" : "/icons/image.gif", "dc:creator" : "Administrator", "ecm:parentId" : "ada5c892-ad99-498d-a919-466deac42ea5", "ecm:ancestorIds" : [ "00000000-0000-0000-0000-000000000000", "e464b908-3607-4ed4-8f23-69b914a00a8e", "2fc5d556-93dc-4392-8901-674de94a7e31", "ada5c892-ad99-498d-a919-466deac42ea5" ], "dc:modified" : ISODate("2017-04-03T19:34:56.635Z"), "ecm:minorVersion" : NumberLong(0), "dc:lastContributor" : "Administrator", "content" : { "data" : "3d4bf419cd8101dcbb3475fa4f7b7138", "name" : "tumblr_ni1rrfEuwn1rsxqqio1_500.gif", "mime-type" : "image/gif", "length" : NumberLong(1962696) }, "ecm:name" : "somethingelse", "ecm:majorVersion" : NumberLong(0), "ecm:lifeCyclePolicy" : "default", "size" : NumberLong(1962696), "dc:created" : ISODate("2017-04-03T19:34:56.043Z"), "dc:title" : "somethingelse", "ecm:primaryType" : "Picture", "ecm:id" : "e55408da-a06b-48aa-814a-ab36395ed16d", "ecm:lifeCycleState" : "project", "ecm:mixinTypes" : [ "Thumbnail" ], "views" : [ { "filename" : "Thumbnail_tumblr_ni1rrfEuwn1rsxqqio1_500.jpg", "width" : NumberLong(83), "description" : "Thumbnail size", "title" : "Thumbnail", "content" : { "data" : "2dadd7007b09362037c9497bbdedb93b", "name" : "Thumbnail_tumblr_ni1rrfEuwn1rsxqqio1_500.jpg", "mime-type" : "image/jpeg", "length" : NumberLong(4912), "digest" : "2dadd7007b09362037c9497bbdedb93b" }, "height" : NumberLong(100), "info" : { "colorSpace" : "sRGB", "depth" : NumberLong(8), "width" : NumberLong(83), "format" : "JPEG", "height" : NumberLong(99) } }, { "filename" : "Small_tumblr_ni1rrfEuwn1rsxqqio1_500.jpg", "width" : NumberLong(468), "description" : "Small size", "title" : "Small", "content" : { "data" : "1e89002b9f3bbee19af3ee7eec801045", "name" : "Small_tumblr_ni1rrfEuwn1rsxqqio1_500.jpg", "mime-type" : "image/jpeg", "length" : NumberLong(154654), "digest" : "1e89002b9f3bbee19af3ee7eec801045" }, "height" : NumberLong(560), "info" : { "colorSpace" : "sRGB", "depth" : NumberLong(8), "width" : NumberLong(468), "format" : "JPEG", "height" : NumberLong(560) } }, { "filename" : "Medium_tumblr_ni1rrfEuwn1rsxqqio1_500.jpg", "width" : NumberLong(495), "description" : "Medium size", "title" : "Medium", "content" : { "data" : "1f7c81b5e56561c43ecd424137ceba50", "name" : "Medium_tumblr_ni1rrfEuwn1rsxqqio1_500.jpg", "mime-type" : "image/jpeg", "length" : NumberLong(173691), "digest" : "1f7c81b5e56561c43ecd424137ceba50" }, "height" : NumberLong(592), "info" : { "colorSpace" : "sRGB", "depth" : NumberLong(8), "width" : NumberLong(495), "format" : "JPEG", "height" : NumberLong(592) } }, { "filename" : "FullHD_tumblr_ni1rrfEuwn1rsxqqio1_500.", "width" : NumberLong(495), "description" : "Full HD size", "title" : "FullHD", "content" : { "data" : "1f7c81b5e56561c43ecd424137ceba50", "name" : "FullHD_tumblr_ni1rrfEuwn1rsxqqio1_500.", "mime-type" : "image/jpeg", "length" : NumberLong(173691) }, "height" : NumberLong(592), "info" : { "colorSpace" : "sRGB", "depth" : NumberLong(8), "width" : NumberLong(495), "format" : "JPEG", "height" : NumberLong(592) } }, { "filename" : "OriginalJpeg_tumblr_ni1rrfEuwn1rsxqqio1_500.", "width" : NumberLong(495), "description" : "Original jpeg image", "title" : "OriginalJpeg", "content" : { "data" : "1f7c81b5e56561c43ecd424137ceba50", "name" : "OriginalJpeg_tumblr_ni1rrfEuwn1rsxqqio1_500.", "mime-type" : "image/jpeg", "length" : NumberLong(173691) }, "height" : NumberLong(592), "info" : { "colorSpace" : "sRGB", "depth" : NumberLong(8), "width" : NumberLong(495), "format" : "JPEG", "height" : NumberLong(592) } } ], "dc:contributors" : [ "Administrator" ], "ecm:fulltextSimple" : "icons image gif administrator administrator 3d4bf419cd8101dcbb3475fa4f7b7138 tumblr ni1rrfeuwn1rsxqqio1 500 gif image gif somethingelse somethingelse thumbnail tumblr ni1rrfeuwn1rsxqqio1 500 jpg thumbnail size thumbnail 2dadd7007b09362037c9497bbdedb93b thumbnail tumblr ni1rrfeuwn1rsxqqio1 500 jpg image jpeg 2dadd7007b09362037c9497bbdedb93b srgb jpeg small tumblr ni1rrfeuwn1rsxqqio1 500 jpg small size small 1e89002b9f3bbee19af3ee7eec801045 small tumblr ni1rrfeuwn1rsxqqio1 500 jpg image jpeg 1e89002b9f3bbee19af3ee7eec801045 srgb jpeg medium tumblr ni1rrfeuwn1rsxqqio1 500 jpg medium size medium 1f7c81b5e56561c43ecd424137ceba50 medium tumblr ni1rrfeuwn1rsxqqio1 500 jpg image jpeg 1f7c81b5e56561c43ecd424137ceba50 srgb jpeg fullhd tumblr ni1rrfeuwn1rsxqqio1 500 full hd size fullhd 1f7c81b5e56561c43ecd424137ceba50 fullhd tumblr ni1rrfeuwn1rsxqqio1 500 image jpeg srgb jpeg originaljpeg tumblr ni1rrfeuwn1rsxqqio1 500 original jpeg image originaljpeg 1f7c81b5e56561c43ecd424137ceba50 originaljpeg tumblr ni1rrfeuwn1rsxqqio1 500 image jpeg srgb jpeg srgb gif administrator", "ecm:fulltextBinary" : "", "ecm:fulltextJobId" : "e55408da-a06b-48aa-814a-ab36395ed16d", "info" : { "colorSpace" : "sRGB", "depth" : NumberLong(8), "width" : NumberLong(495), "format" : "GIF", "height" : NumberLong(592) } }
mongos>
However I never see the s3 bucket being leveraged for store.
