Batch import of metadata / aspects on Alfresco?

I'd suggest using a Web Script (http://wiki.alfresco.com/wiki/Web_Scripts) for uploading the metadata. Your batch import process would upload files as usual, then call a Web Script to set the metadata on those files.

Cheers,
Peter

mdutoo

Hi

Beyond webscripted import as pmonks said, you can use the Open Source ETL tool Talend 3.1 and its Alfresco connector to import data (taken e.g. from XLS, XML, databases or anything Talend can look into) into Alfresco as any kind of Alfresco content (documents, folders, custom…).

Otherwise there is an ACP generator (but I didn't test it) or you could even develop a specific automated action, though that probably would amount to more work.

Some links :
Talend 3.1 including the Alfresco connector is available at http://www.talend.com/download.php
Its Alfresco connector has been contributed on the Alfresco forge at http://forge.alfresco.com/projects/etlconnector/
More information at http://knowledge.openwide.fr/bin/view/Main/AlfrescoETLConnector

That reminds me - OpenMigrate (http://www.tsgrp.com/Open_Source/OpenMigrate/open-migrate.jsp) does something like this too. One of my colleagues used it quite extensively for a Hummingbird -> Alfresco migration and thought it was pretty good.

Cheers,
Peter

mdutoo

Interesting, I didn't know about it, thanks for the feedback.

Hi Mdutoo, thank you for the information. About Talend Open Studio, it is an open source ETL tool for data integration experts with a user-friendly GUI, easy to learn for a non-technical user. What distinguishes Talend, when it comes to business users, is the tMap component which allows the user to get a graphical and functional view of integration processes. For more information: http://www.talend.com/

trelofysikos

thanks for the feedback guys.

I gave talend a wuick try and thought it was a bit confusing. If anyone can help me with how to import a certain tree structure with metadata in alfresco i would be gratefull.

Also i'm now downloading openMigrate which seems promising. Anyone ever used this for batch import of documents + metadata in alfresco ?

Hello trelofysikos,

With Talend you can perform many functions and if you are trying to load metadata in Alfresco, you can perform that operation easily. Talend has connectors to Alfresco so you can import metadata from Alfresco.
About importing a tree structure, you should be able to do that with Talend.

If you have any questions, feel free to ask, we will answer the best we can.

itbeb

What does one do if you want to migrate ordinary Windows file system docs to Alfresco and want to keep the dates in tact, i.e. you want the created and modified dates to stay the same when you upload it to Alfresco and not to change to the current date?

trelofysikos

Hello trelofysikos,

With Talend you can perform many functions and if you are trying to load metadata in Alfresco, you can perform that operation easily. Talend has connectors to Alfresco so you can import metadata from Alfresco.
About importing a tree structure, you should be able to do that with Talend.

If you have any questions, feel free to ask, we will answer the best we can.

Hey Groutal,

can you give me a simple example of one file+metadata import to alfresco using Talend ?
like a step by step tutorial ?

Hey,

Absolutely! Check out the Talend tutorials available on the website: http://www.talendforge.org/tutorials/menu.php

bwakkie

ps as far as I understood does Talend only support alfresco version 2.x if I read this correctly.

You might want to check the component page on the Talendforge website. The latest update on the component was made on Talend version 3.1.

Component page:
http://www.talendforge.org/components/index.php

freebsdboy

This is rather strange.. the process and hoops to go through to migrate data. Anyone who deploys Alfresco needs to do this.

There is no prebuild acp file creator? or no one has shared theirs?

I'm walking through the steps to create the acp or metadata extract using Talend, a very cool product.
Using tutorial http://www.talendforge.org/tutorials/tutorial.php?idTuto=3#step53

But the turotial is generic. So what creates the import file on a Linux server?
a good old ls -l /tmp/importmetadata.file
or
find . -print | xargs ls -gilds >/tmp/importmetatdata.file

rliu

freebsdboy,

You are absolutely correct. Most customers using Alfresco will most likely be migrating documents, metadata, content, etc. from an existing CMS or some form of document management system. My approach was to re-engineer the exported Alfresco Content Package (ACP):

1. Define the content model as needed.
2. Deploy to environment where you can create some test content or upload a document of some kind that would be representative of the migrated content.
3. Do an export of your test data.
4. Review the schema from the XML that was generated from the export.
5. Create a program that can generate an XML that reflects the XML file that was exported.
6. Package into an ACP.
7. Import into Alfresco.

This was not the easiest of solutions, but I think it was a good approach. I wrote a program that generated the XML and other content items. Build a stable repeatable process along the way as it'll require a lot of patience and testing. My ACP file that I used to import worked in various Alfresco Labs 3.0 stack. I was able to package 600 documents in its respective space. Though, 600 was all I had, I believe this approach can be repeated for larger volume.

Best of luck!

freebsdboy

Care to share the work you've done?

Hey this is open source - we all shouldn't have to redo the same thing.

It's not an all-singing, all-dancing content migration tool (OpenMigrate is a better choice if that's the requirement), but I've put a simple filesystem import process into Google Code at http://code.google.com/p/alfresco-bulk-filesystem-import/. This package is intended to simply and efficiently handle the basic use case of importing a large number or size of files & folders from local disk into Alfresco.

Please take the time to read the readme file [1] before using it. The current version is still under heavy development and has a number of major functional gaps (eg. no support for loading metadata yet, although that work is in progress) that will preclude it for specific use cases, but for the basic case of quickly loading in a large number of assets it already works well.

Cheers,
Peter

[1] http://code.google.com/p/alfresco-bulk-filesystem-import/source/browse/trunk/README.txt

norgan

Hi Rich liu,
what where the major problems you had to solve ? Would this offer itself to be written as to be so generic to be of public use ?

i think about something like XSD transformation or atleast a XML defintion of the dataformat, which is to be imported (worst case - from an excel file or CSV file) Maybe this could go with Talend or become an alfresco forge project ?

Regards, Norgan

rliu

It'll take a little time to generify the code and make it extensible. As soon as I have something I feel worthy of publishing, I will provide it to the community. In the meantime, the actions taken as I described should allow you to migrate data into Alfresco (inclusive of custom properties).

ajkalbente

Inform us on what is going on with the project! Thanks.

stevegreenbaum

The latest information regarding the Talend/Alfresco connector's support for Alfresco v3.3 is posted at http://forge.alfresco.com/forum/forum.php?thread_id=1274&forum_id=752

Hello,

I have some data to import to alfresco.
Your method seems right.
Could you send me your source code so that i can adapt it to import my documents to alfresco

Thank you

nramdoo, the link to the Google code project (which includes both compiled binaries and source code) for the bulk filesystem import tool is listed above.

Cheers,
peter

hello,

How i solved my imports ?

I used Alfresco 3 web services bought from PACKT publishing.
I have been able to import into the alfresco repository using webservices and java and the Chapter 2 "Creating and modifying contents" greatly helped me.
I also used the following links
http://dev.alfresco.com/resource/docs/java/web-service-client/org/alfresco/webservice/types/package-summary.html
At first it may seem difficult, finally it was not so difficult , but i used java language.

I don't think that it is the best method ? Next time i will try to use webscripts.

Thanks

stevegreenbaum

As I understand it, the Talend connectors are out-of-date and won't work with v3.3 forward. Have you used the connector with v3.3 or v3.4?

Hello,
I used alfresco 3.3G.
I am able to create spaces, load content (images, pdf html)
I also made mass update of properties.
At first, i tried to understand the talend connector. I am quite sure (confirmed by many messages on some forums) that it no longer works with alfresco 3.3
Bye

You may also want to evaluate the "Bulk Filesystem Import Tool" [1]. It's been used in several production Alfresco instances (that I'm directly aware of), and has seen quite a bit of traction amongst Alfresco's partner community as well.

Unlike the ACP mechanism, the approach the BFSIT takes requires minimal reformulation of the source content and is significantly faster as well (it's been tested well into the hundreds-of-thousands-of-documents / tens-of-gigabytes range, and is designed to work on virtually any content set, no matter how large).

Cheers,
Peter

[1] http://code.google.com/p/alfresco-bulk-filesystem-import/

Hello pmonks,

I will use your program the next time i will have to do a bulk import.

best regards
Nr