APACHE HOP (ETL)

The latest version is [0.70] (April 2021). It seems like a promising ETL tool.

Links:

(Latest) Main features:

  • Dark Mode
    [Menu] Tools > Look & Feel Tab, click on ‘Dark Mode’ checkbox. Restart
  • Kettle Importer!
    File > Import from Kettle/PDI
    Browse to your Kettle folder for a .ktr file
    Select in field ‘Import in Project’: default
    Click on ‘import’
    Open your workflow that should be on your ‘./hop’ folder as .hpl files
  • Git Diff on the HOP Gui

To run your own HOP:

  1. Use a Docker container:
    $ docker run apache/incubator-hop
  2. Or install HOP
    Requirements: java 8, Java 11

Start it with:

./hop/hop-gui.sh

  • To see an example:
    File > Open
    Browse to: ./hop/config/projects/samples/transforms
    Double click on on a file
  • To start a new ‘workflow’:
    File > New > Pipeline
    File > New > Workflow
  • The following is a Kettle transformation migration to a HOP pipeline:

Notes:

  • It feels weird thatwindow dialogs are open on any click, it takes time to get used to.
  • You can still edit the .hpl as an XML file like Kettle/PDI 🙂
  • Files are stored in the default folder:
    \hop\config\projects\default
  • In a transformation import, I lost fields information on Excel Step. It might be that config pages are set differently.

Advertisement

VI. Case Management Example

To document the functionality of the Mayan EDMS application we will test the simple Workflow we have set up in the previous posts and show the steps and strings the users would perform taking the document through the following States.

StateActivitY
ReceptionThe user verifies the physical claim, scans the document as a PDF and uploads the files to Mayan EDMS
AnalystThe user would verify eligibility and ad supporting documentation.
Then send that info for authorization
ExpertA user that could authorize payment would review the documents and reject it at this time as it needs historical data.
Note: there is a ‘Smart Link’ functionality so we could associate all claims maybe using the ‘insurance policy’. We haven’t done that set up but is similar to the index set up.
AnalystAdds the requested information (using clinical history PDF) and set it again for approval
ExpertThis time we assume the approval is done, payment and notification is done.
Note: There is an example in the forum to send an automatic email.
VaultThe document will reside in an historical vault.

The Analyst and Expert states have a corresponding group and ‘Tags’ so the users can locate the documents easily. The first and last States have documents that can be accessed to this group but using the tags ‘Claim review’ and ‘Expert’ would be easier for them. It also make useless to notify users by mail and flood their incoming folder with spam.

The workflow with the States, Transitions and Tags in a diagram is


I. Upload

To start the exercise we will log in as root and impersonate ([System-Action] > Tools > Impersonate user]) or log in as each of them

Log in as jsmith/Password54321 to receive a document and upload it: [Left-Menu] > Documents > New document, then

  • Choose the document type
  • Type the metadata: Select ‘Department A’, 20210511 as Claim ID, 05/11/2021 as date received.
    Important: Response (ID) will be left empty and filled at the last step so we could search for the ID as we open the request and the solution ID.
  • The tag will be left empty as we have setting and removal of them on automation
  • Then choose our ‘Claim PDF’ and click on the ‘Document received’ button
  • List the recently added document:
    [Left-Menu] > Documents > Recently added

II. Review our document

  • Locate it with
    [Left-Menu] > Documents > Recently added
  • Click the link to preview the document
  • Review options:
    [Action-Select] > OCR
    [Action-Select] > Pages
    [Action-Select] > Metadata

III. Transition to next state

  • In the document page click
    [Action-Select] > Workflows
  • Then click the ‘Transition’ button
  • Select the next state (we only have one as the workflow)
  • Add a message and click the ‘Submit’ button

IV. Analysis

This step is the first state that has a Tag associated with it. That will make easy to locate documents that need attention in a specific state

Log in as adow/Password54321 and:

  • Open the document
    Check the document name, confirm that the cryptic name is now the ID as we configured in automation.
  • Open documents with tag ‘On Claim Review’
    [Left-Menu] > Tags > All
    Click on Documents button
  • You can download any version of a document by
    [Action-Select] > Versions
    Click the ‘Download version’ button
  • Next upload a document that will be needed in the next stage
    [Action-Select] > Versions
    [Action-Select] > Upload version
    Add a comment, select the file and click the ‘Upload document’ button on the far bottom-right
  • Check again the versions, there will be now two of them one with 2 pages the new one with five. The OCR option will show the latest one.
  • Now do the transition to ‘Expert’ as we did in last step

V. Expert

This stage has two transitions, the one that will be shown is the one that takes the document back in the workflow. We will ask for more information.

Log in as the Expert hbrown/Password54321 or impersonate it from the admin user:

  • Open documents with tag ‘On Expert’
    [Left-Menu] > Tags > All
    Click on Documents button
  • Open the document
  • Open the workflow and choose the transition ‘Reject’ to send the document back with a comment that more information is needed (clinical report).

VI. back in Analysis

Log in as adow/Password54321 or impersonate the user:

  • Check the log to check the last message.
    Open documents with tag ‘On Claim Review’
    [Left-Menu] > Tags > All
    Click on Documents button
    [Action-Select] > Workflow
    Click the ‘Detail’ button
  • Add a new version with the requested file
  • Make the transition to the Expert

VII. Expert Again

This time the document will advance in the process.

Log in as the Expert hbrown/Password54321 or impersonate it from the admin user:

  • Check that the document is Tagged ‘On Expert’
  • Open versions and check that we have al three of them
    We can preview and download al documents needed to process this request even add a payment document but we have done an upload already.
  • As the process will be done we add the final ID.
    But the metadata page only shows the items with data so we need to ‘add’ a metadata ‘4. Response’ as ‘PAIDCLAIM20210515’
  • Then transition to ‘Vault’ with a note.
    The whole log will be shown
  • Check the Tags to confirm that there is no document to be processed

VIII. Historical Vault

The process is completed but we need a repository of our cases, so in this example there is a state that is out of the tagged States that holds all of them.

Log in as jsmith/Password54321 but all of our users have been configured to have access to this Document type.

  • Locate the document in the ‘Vault’ state it using the Indexes:
    [Left-Menu] > Claims > Department A > 2021-05
  • Also check the states to verify that the document is in the last one

IX. Search

The search field is in the main page (click the Mayan logo) to get there.

We can search for a metadata value (Claim ID=2021 or Response=PAIDCLAIM2021051), full text (‘country context’ or ‘what is this form’) in different versions. The result page will show the document only not the exact page. If it is not found it will show the documents it searched with an “X” to signal it is not in that document.

That’s it the hole workflow route.

V. Users, Groups, Roles & ACL

To show how roles and ACL’s can be used to simplify Mayan users UX (users experience) we will create two groups. One will be for people uploading and supervising the document flow and another with the users that actually do work with the documents.

I. Users

As for users the ones to be used are for:

  • Control Desk – Will receive and upload the documents
  • Analyst – generates the response to the claim
  • Expert – authorizes or rejects the claim

How To

  • [System-Select] > Setup
  • Click ‘Users’
  • Click on ‘Create new user’ or
    [Action-Select] > Create new user
  • Type (for each user)
    Username, First Name, Last Name, email
    Click ‘Save’
  • In the next screen type twice the password you like. It could be the same for all users in this test. (I use Password54321).
    Click ‘Submit’
    Note: You can enforce different levels of complexity for your passwords in [System-Select] > Setup, Click ‘Settings’ then ‘Django’ button and set ‘AUTH_PASSWORD_VALIDATORS’.
UsernameFIRST NameLAST nameEmail
jsmithJamesSmithjsmith@urexample.com
adowAnnaDowadow@urexample.com
hbrownHectorBrownhbrown@urexample.com

II. Groups

As we said we will use two groups.

How To

  • [System-Select] > Setup
  • Click ‘Groups’
  • Click on ‘Create new group’ or
    [Action-Select] > Create new group
  • Type (for each user)
    Name
    Click ‘Save’

The groups are:

  • Control Desk
  • Workgroup

Associate Users to Groups

How To

  • [System-Select] > Setup
  • Click ‘Groups’
  • On each of the [Item Options] click ‘Users’ Button
    Select from the left panel the ‘Available User’
    Click ‘+Add’
  • To return to the Groups page:
    [Acion-Select] > Groups
Group [Item Option]UserFunction in test
Control DeskjsmithReceiver
Workgroupadow
hbrown
Analyst
Expert

III. Access Control : Role

Mayan posses a Rol-Based Access Control (RBAC) security approach to configure system privileges to users.

How to

  1. Rol‘ granted permissions allow system wide access.
    For our ‘Control Desk’ group we use a ‘Rol’ with all the permissions will be assigned, except for deleting files, emptying the trash and impersonating other users.
  2. For our two ‘Workgroup’ we use ‘Access Control List (ACL)’, we can select items (Workflows, States, Documents, Indexes, Cabinets, Tags) and then choose the group and permissions specific for that object.

How-To [Roles]

  • [System-Select] > Setup
  • Click ‘Roles’
  • Click on the button ‘Create new role’ or
    [Action-Select] > ‘Create new role’
  • Type:
    Label: Control Desk role
    Click ‘Save’

Repeat for ‘Workgroup Role’.

Now associate our roles to our groups

  • [System-Select] > Setup
  • Click ‘Roles’
  • On ‘Control Desk Role’ [Item Option] click ‘Groups’ button
  • On the left panel select ‘Control Desk’
    Click ‘+Add’
  • To return to the role page
    [Action-Select] > Roles

Then repeat the steps but assign:
– ‘Workflow Role’ to group: ‘Workflow’


After setting the permissions that we will do in the following paragraphs for the group that acts like a ‘supervisor’ the Roles page looks like this:

  • [System-Select] > Setup
  • Click ‘Roles’
  • On the ‘Control Desk’ [Item Option] click on ‘Role permissions’
  • Click on the ‘+Add all’ button
  • Now browse on the right panel for each of the following permissions and click on ‘Remove’ button:
    • Documents > Delete documents
    • Documents > Empty trash
    • Documents > Restore trashed documents
    • Authentication > Impersonate users

Those actions should be done by the administrator or an special account. In that way you browsed over many permissions that the system offers.

How-To [ACL]

For ACL our Workgroup will need permissions to see ‘Document types’, ‘Workflows’, ‘indexes’, ‘Tags’.

First associate our ‘Claim’ document type just to the ‘Workflow 1’:

  • [System-Select] > Setup
  • Click ‘Document Types’
  • On the ‘Claims’ [Item Option] click on the ‘ACLs’ button
  • Click on ‘New ACL’ or
    [Action-Select] > New ACL
    Select ‘Workflow Role’
    Click ‘Save’
  • On the ‘Access control list for: Claim’ page
    And on the ‘Workflow Role’ [Item Option] click on the ‘Permissions’ button
  • On the “Role Workflow Role permissions for ‘Clam'”
    Click ‘+Add All
    Then to reduce de buttons available to users, select from the right panel and click ‘Remove’:
    • Common > Copy object
    • Converter > Create new transformations
    • Converter > Delete transformations
    • Converter > Edit transformations
    • Document types > Delete document types
    • Document types > Edit document types
    • Documents > Delete documents

It should look like this:

Now will be setting special permissions for the ‘Workflow’ group.

So it can see the workflow:

  • System-Select] > Setup
  • Click ‘Workflows’
  • On the [Item Option] ‘Claims Workflow’ click on the ‘LCAs’ button
  • Click on ‘New LCA’ or
    [Action-Select] >
    Select ‘Workflow Role’
    Click ‘Save’
  • On the left panel select three available permissions:
    Document Workflows > Execute workflows tools
    Document Workflows > Transition workflows
    Document Workflows > View workflows
    Document Workflows > Transition workflow instances [New in version 4]
    Click ‘+Add’

So it can see indexes:

  • [System-Select] > Setup
  • Click ‘Indexes’
  • On the [Item Option] ‘Claims’ click on the ‘LCAs’ button
  • Click on ‘New LCA’
    Select ‘Workflow Role’
    Click ‘Save’
  • On the left panel select
    Indexes > View document index instances
    Indexes > View document indexes’
    Click ‘+Add’

And to see the Tags:

  • [left-menu] > Tags
  • Click ‘All’
  • On the [Item Option] ‘On Claim Review’ click on the ‘LCAs’ button
  • Click on ‘New ALC’ button
    Select ‘Workflow Role’
    Click ‘Save’
  • On the left panel select
    Tags > View tags
    Click ‘+Add’
  • [left-menu] > Tags
  • Click ‘All’
  • On the [Item Option] ‘On Expert’ click on the ‘LCAs’ button
  • Click on ‘New ALC’ button
    Select ‘Workflow Role 1’
    Click ‘Save’
  • On the left panel select
    Tags > View tags
    Click ‘+Add’

That is for our example but now you can assign permission to access just a State or documents in that state/tag. With another document type you can work with the same workflow or another one special documents that the other workgroups wont see. And with transitions you can even change privileges (ACL) or even send emails or ask Mayan or other systems to to perform actions via their API.