Traversing Nested Data Structures with DataWeave
The problem…
A friend of mine sent a request on Slack today, he asked if anyone had tried to solve a particular problem he had found posted on twitter. The post references this GitHub page where the problem and some of the solutions are described. My friend specifically asked if anyone had solved this with pure DataWeave, a transformation language embedded in the MuleSoft software known as Mule.
I haven't coded anything in DataWeave for quite a while, so I thought I'd give it a go. I had to read up on some of the functions, and then use this REPL to do the work. I copied the example input from the GitHub page, and started hacking away.
A first pass…
At first I thought I could use some nifty recursion and a “stream” data structure. Basically, that is a pair, the first part is the current value, the second is a function which returns a pair with the next value and a function. That didn't work out because, well, recursion. DataWeave is not tail-call optimized and will overflow the stack at 256 frames.
Bummer.
A second pass…
Okay, so looking at it from a different perspective, I just needed to setup the indices correctly and then I could iterate over the payload and come up with a solution.
%dw 2.0
output application/json
var allWithReset = zip(payload map $.reset_lesson_position, payload.lessons map $$ )
var allStartVals = allWithReset map if ($[0] == true) 0 else $[1] + $[1]
---
payload map (value, index) -> {
title: value.title,
reset_lesson_position: value.reset_lesson_position,
position: index + 1,
lessons: value.lessons map {
name: $.name,
pos: allStartVals[index] + ($$ + 1)
}
}
That did the trick… with the 3 records in the example data set. But I decided to add another record. And, well, you can guess what happened next… yep, the indices were off – completely. Obviously, the problem is how I was creating the indices, so let's look there for a solution:
%dw 2.0
output application/json
var allWithReset = zip(payload map $.reset_lesson_position, payload.lessons map $$ )
var allStartVals = allWithReset map if ($[0] == true) 0 else $[1] + $[1]
var allStartNormalized = allStartVals map if (allStartVals[$$ - 1] == 0) 2 else allStartVals[$$]
---
payload map (value, index) -> {
title: value.title,
reset_lesson_position: value.reset_lesson_position,
position: index + 1,
lessons: value.lessons map {
name: $.name,
pos: allStartNormalized[index] + ($$ + 1)
}
}
Yep, that fixed it! Well, at least until I added another record and changed one of the reset values… And then the indices are broken again. So, the solution is just to add more complexity, right!??!
%dw 2.0
output application/json
var allWithReset = zip(payload map $.reset_lesson_position, payload.lessons map $$ )
var allStartVals = allWithReset map if ($[0] == true) 0 else $[1] + $[1]
var allStartNormalized = allStartVals map
if (allStartVals[$$ - 1] == 0 and allStartVals[$$] == 0)
0
else if (allStartVals[$$ - 1] == 0 and allStartVals[$$] != 0)
2
else allStartVals[$$]
---
payload map (value, index) -> {
title: value.title,
reset_lesson_position: value.reset_lesson_position,
position: index + 1,
lessons: value.lessons map {
name: $.name,
pos: allStartNormalized[index] + ($$ + 1)
}
}
Nope, that fails too. At this point I start getting a little banter in the slack channel. We discuss the merits of different approaches to a solution, then one of my friends posts a working solution that is elegant and, after thinking about it, somewhat obvious. As a matter of fact, after saying recursion several times, I realized I needed to be reaching for the reduce
function, which is how DataWeave manages recursion.
The solution…
Nope, not going to post it here, you'll have to go read this blog post to get the answer. This guy writes some nice blog posts as well, so feel free to peruse and learn a few things. Good stuff!!
Tags: #MuleSoft #Dataweave