Be Like DerManu
A recent (March 2025) work assignment of mine was to write a bespoke data-wrangling QGIS plugin for a client. The backbone of the plugin is PyQt5, and this project was my first exposure to it and to GUI programming. It was a treat: lots to learn, lots to explore, lots to break, lots to ponder.
This short article describes a bug fix which resulted in a candidate for the shortest commit, at just two bytes, that I have pushed in my career thus far, and considers the value of actively maintained developer forums in the age of AI.
Context
A central component in my plugin is a data table that displays the
results of a user’s query. Users are able to check a checkbox to select
displayed records for further processing. Data is stored in a data model
object, which is subclassed from Qt’s QAbstractTableModel
class, and display is handled by a QTableView object, to
which the model is attached. The model and the view interact, as most
everything else in QT does, via a signals-and-slots mechanism: objects
that change state emit signals, and objects that need to be notified of
these changes receive their notifications via slots.
self.links_model = MyLinksTableModel(links_, self)
self.links_table.setModel(self.links_model)
SIGNAL: SLOT:
"new $thing, . "got it, thanks!"
coming in hot!" . . /
\ . . . /
\ . . . . /
v . . . . . v
+--------------+ . . . . . +--------------+
| >>> . . . . . . _} |
| | . . . . . | |
| DATA MODEL | . . . . . | DATA VIEW |
| | . . . . . | |
| | | |
+--------------+ +--------------+Into the Pan
When I implemented the model-view interaction, I was stymied by a
bug: when clicking my Toggle Select button, nothing would
happen, until I panned my cursor atop the data view: only then would the
checkboxes update visually. This was the problem that helped me begin to
conceptualize the event-driven architecture of Qt. My data model was
changing, but it was not emitting the correct signal. Custom functions
and method overrides implemented in the data model class need to
explicitly emit the appropriate signals. In this case, I had a button
that, when clicked, toggled the selection checkbox for all displayed
records. The function that implements this behaviour is pretty simple:
calculate the indexes for the column of checkboxes, call setData() for
each index, and then at iteration’s end, emit the dataChanged signal for
all changes. Because I had overridden the setData() method
of the model class, I needed to add the following line to my toggle
function:
self.dataChanged.emit(self.index(0, 0), self.index(self.rowCount(), 0))and with that, all was well: click the Toggle Select
button, and boom, all my little checkboxes dutifully checked or
unchecked themselves.
Out of the Pan, Into the Fire
A short while later, I was asked to implement sorting in the data
display table. To do so, a QSortFilterProxyModel was
required. It’s positioned like so:
SIGNAL: SLOT: SIGNAL: SLOT:
"new $thing, . "got it! ... forwarding!" . "got it, thanks!"
coming in hot!" . . \ / . . /
\ . . \ / . . /
\ . . . \ / . . . /
v . . . . v v . . . . v
+--------------+ . . . . +---------------+ . . . . +--------------+
| >>> . . . . . _}-->-->-->-->-->>> . . . . . _} |
| | . . . . | | . . . . | |
| DATA MODEL | . . . . | PROXY MODEL | . . . . | DATA VIEW |
| | . . . . | | . . . . | |
| | | | | |
+--------------+ +---------------+ +--------------+ Implementing the proxy model was deceptively simple, just three lines of code between the model initialization and linking the model to the table view:
self.links_model = MyLinksTableModel(links_, self)
self.links_sorting_proxy = QtCore.QSortFilterProxyModel(self) // new
self.links_sorting_proxy.setDynamicSortFilter(True) // new
self.links_sorting_proxy.setSourceModel(self.links_model) // new
self.links_table.setModel(self.links_sorting_proxy)I thought I was coasting, but then the other shoe dropped: the toggle-selection visual delay reappeared with a vengeance. The table view refused to show the checkboxes’ changed state until some other activity happened to the view to register an update, e.g. my cursor panning across it. But my signal is being manually emitted!! What’s going on now?
Saved by DerManu
I was so, so lucky with this one. I’d been tearing into my code for a couple of hours and was growing increasingly frustrated. I was searching for every question permutation and tried all sorts of suggestions but nothing helped. Then, I found it: a Qt Forum post dated February 17, 2012, from user DerManu. Their post described my exact situation, and my exact problem, and bless their sweet, sweet soul: their next post, 7.5 hours after the first, described the fix! and it worked!!!
Here is their solution post:
Ha! It’s always the things that “are correct for sure” that bite you! Turns out just printing “Yey!” wasn’t so smart, the parameters I passed to dataChanged were wrong and so the filter proxy didn’t let the signal through.
I emitted like this:
@emit dataChanged(index(id, 0), index(id, columnCount()));@
When I should have emitted
@emit dataChanged(index(id, 0), index(id, columnCount()-1));@
Obviously ;) Sadly this “column out of bounds” situation doesn’t produce any qDebug output in the model or view, but is silently turned into column id -1 and row id 0 (!). Sneaky.
… sure enough, my Python version had the exact same off-by-one error. So easy to miss, so easy to fix: just two bytes, and everything is grand again. Behold, my shortest commit to date:
BUG FIX: signals emission relay from links model through proxy
The dataChanged signal must be emitted from any custom function that modifies model data so that its correlated
view can be updated. Previously, this signal emission was working. However, once the sorting proxy model was put
in place, the signal failed to propagate through the proxy and back to the links table view.
The reason is that the proxy model class applies a set of checks on its inputs, and reacts differently depending
on the checks; for some, it sends output to QDebug (if implemented), or raises an exception. For others, it silently
adjusts values and carries on its merry way like a sneaky little goblin.
Prior to implementing the proxy model, the dataChanged signal emit code attempted to capture the full width and
depth of the table by referencing self.rowCount(). It should have been self.rowCount() - 1 , but this error was
silently ignored - until it was passed through the proxy sorting model, where it was silently _changed_. This resulted
in the select-all button correctly selecting all, but not updating the view until a mouse event was registered over the view.
With this two-byte-long commit, we now have a working - and correct - signals emission for the function
links_data_model.toggle_checkbox_selections().
Credit to user DerManu circa Feb 17, 2012, at the Qt Forum: https://forum.qt.io/topic/14175/subclassed-qsortfilterproxymodel-doesn-t-forward-datachanged-signal
If I have the privilege to meet you one day, DerManu, your first beer is on me.
71 - self.dataChanged.emit(self.index(0, 0), self.index(self.rowCount(), 0))
71 + self.dataChanged.emit(self.index(0, 0), self.index(self.rowCount()-1, 0))A Bit of Introspection
With this fix, I was elated. An eldritch wizard, having walked these corridors years prior, had left etchings on the walls for future explorers, and my torchlight had just caught the glint of this particular bit of wisdom, allowing me to progress to the next chamber. I felt connected to a broader community of people out there solving problems and helping each other along the way.
This short story-arc - encountering a problem, troubleshooting all sorts of almost-but-not-quite solutions, finding the solution shared by another - tapped into a sentiment, a deep-gut feeling, that has been growing in the corners of my mind but still feels a bit vague, a feeling which I am still struggling to define and communicate. Maybe a bit of personal context will help give it shape.
I am a late inductee to the software development community, having spent the first twenty years of my adult life working as a geologist. I made the career switch because, after being introduced to Python by a colleague some time around 2017, I fell head-over-heels in love with programming: the problem-solving, the deeply technical nature of the work, the weird and arcane lore surrounding it all. Very early in my journey of teaching myself how to program, I took to heart the advice of a friend: turn off all the helpful hints and learn how things work under the hood. I avoided IDEs like the plague, and in my beloved Sublime Text editor I turned off everything except for syntax highlighting. With every new language, tool, and concept I spent hours crawling down rabbitholes, spelunking down through the layers of abstraction. I spent - and still spend - countless hours reading textbooks and long-form blog posts and archived mailing lists and Stack Overflow answers and everything in between. I chewed through books like Where Wizards Stay Up Late, The Cuckoo’s Egg, Code, The Cathedral and the Bazaar, and many more. I even worked my way through Nand2Tetris on paper, drawing out all the diagrams and writing the HDL in a moleskine book while convalescing in Mexico for six months. My learning journey has been slow going - there is a reason why IDEs exist, of course, and trying to find an answer in documentation when you don’t understand the question you’re asking is a challenging task. But I have loved every minute of it, and I’m just as excited about it all now, as I was when I first wrote Hello World in Python. Now that I have actually made the big career Leap of Faith and landed a job as a software developer, I’m happy to have adopted a bit more assistance in my dev environment. Pyright has proven to be a huge boon - the documentation I need is right there at my fingertips. I still keep intellisense and pop-up suggestions turned off, though; I hate how they grab my attention. All those tools’ help is instead tied to keyboard shortcuts - it’s on demand under my control, rather than jumping out at me unwanted.
Continuing in this vein, I have been avoiding AI tools like the plague. Something just feels off about it. I don’t feel great about pushing a solution generated by AI assistance. I don’t feel bad about it, but something - the magic - that gives me joy when I solve a problem and implement a solution, is missing when AI is involved. I feel like the developer community is suffering as we adapt to the new AI landscape. The genie is most certainly out of the bottle, and there’s no stuffing it back in; we’ve no choice but to adapt. There are, however, ways of going about this transition that are healthy, and ways that are not. We should never lose sight of the fact that we people, whether software developers or any other group of people, constitute a community, and that communities require active care and maintenance.
I must acknowledge here that for many people like myself, programming is a passion, a hobby, a thing of interest. For many others, it’s just a means to an end. I suppose that, in this light, it’s fair that the community experiences a bit of shrinkage. Those who remain, remain by choice, and those who hold other interests can be free to pursue them. That is an excellent benefit that AI can deliver: like many inventions before it, it can offer us a higher degree of freedom, to choose where we spend our limited time and efforts. Just like those many other inventions, optimizing the degree of freedom, and ensuring that we actually do get to make new choices, requires a deliberate understanding of what we want to achieve as a community, and active effort working toward those goals.
Software development, like any other career pursuit, can become completely emotionally crippling under the right circumstances, for example when the motives of employer and employee are too far misaligned. It can also be a wonderful complement to our lives, as both a pursuit and a tool. With the advent of AI, we must be vigilant, mindful, and conscious of what we stand to gain and lose as we lurch and stumble awkwardly toward new horizons.
I suppose that, ultimately, my motive for writing this post is to share how Great I felt when I stumbled across this solution on a forum, knowing that DerManu had posted their solution over a decade prior, ready for someone just like me to come stumbling along in need of a bit of help. So to those of you out there solving problems, I urge you to be like DerManu: keep sharing these small wins! Update your forum questions with answers, if/once they are found. Record your hard-earned wisdom for those who may one day follow in your footsteps. It’s a time-honoured and battle-tested way of passing knowledge on to the next generation and helping people to build onward and upward. And don’t let AI suck all the fun out of this wonderful and weird pursuit.
Down the Rabbit-Hole
How did my bug come to be in the first place? What internal Qt mechanisms precipitated this issue? Here I do a bit of code spelunking. I should note that I have not found any sort of root cause: I had to stop this adventure because I have (1) reached the limit of my knowledge with C++, and (2) too many other things have been grabbing at my attention and I needed to do some triage. Nonetheless I wanted to at least post some progress along with my musings. Perhaps a part II will be warranted, if one day I set up a debugging environment and start digging in earnest. Don’t hold your breath :)
In any case. Recall the sequence of events: first, the data model -
table view connection was established, and everything worked fine. When
the toggle-all-checkboxes functionality was added, the problem first
arose, but then was quickly squashed by emitting the
dataChanged() signal from the toggle method, despite
the off-by-one error. Subsequently, the proxy model was
implemented, which resulted in the same userland symptoms and ultimately
exposed the off-by-one error in the dataChanged.emit()
logic.
When the model-view connection was direct, the off-by-one error was
being ignored or bypassed somehow, whereas when the proxy was put in
place, the off-by-one error was being caught. This implies that we
should be looking for clues as to how the table view handles the
dataChanged() signal versus how the same signal is handled
by the proxy model.
QAbstractItemView(QTableView)
NB: for all the code below, the isValid() method is a
check on QModelIndex objects; it’s an inline bool defined in the header
file for QAbstractItemModel:
Q_DECL_CONSTEXPR inline bool isValid() const noexcept { return (r >= 0) && (c >= 0) && (m != nullptr); }where r is row, c is column, and m is model. I don’t think the problem is in there.
The table view is a QTableView
object, which inherits from QAbstractItemView. A quick scroll through the object's methods shows that the
dataChanged() slot method is implemented in the parent
QAbstractItemView class, so that’s the entry point for this
particular burrow. The first code block below shows the
dataChanged() slot method. It takes two
QModelIndexes as parameters; these, as their names suggest,
define the top-left and bottom-right (x,y) (inclusive)
corners of the chunk of the data view that we want to change.
3339 /*!
3340 This slot is called when items with the given \a roles are changed in the
3341 model. The changed items are those from \a topLeft to \a bottomRight
3342 inclusive. If just one item is changed \a topLeft == \a bottomRight.
3343
3344 The \a roles which have been changed can either be an empty container (meaning everything
3345 has changed), or a non-empty container with the subset of roles which have changed.
3346
3347 \note: Qt::ToolTipRole is not honored by dataChanged() in the views provided by Qt.
3348 */
3349 void QAbstractItemView::dataChanged(const QModelIndex &topLeft, const QModelIndex &bottomRight, const QVector<int> &roles)
3350 {
3351 Q_UNUSED(roles);
3352 // Single item changed
3353 Q_D(QAbstractItemView);
3354 if (topLeft == bottomRight && topLeft.isValid()) {
3355 const QEditorInfo &editorInfo = d->editorForIndex(topLeft);
3356 //we don't update the edit data if it is static
3357 if (!editorInfo.isStatic && editorInfo.widget) {
3358 QAbstractItemDelegate *delegate = d->delegateForIndex(topLeft);
3359 if (delegate) {
3360 delegate->setEditorData(editorInfo.widget.data(), topLeft);
3361 }
3362 }
3363 if (isVisible() && !d->delayedPendingLayout) {
3364 // otherwise the items will be update later anyway
3365 update(topLeft);
3366 }
3367 } else {
3368 d->updateEditorData(topLeft, bottomRight);
3369 if (isVisible() && !d->delayedPendingLayout)
3370 d->viewport->update();
3371 }The dataChanged method first checks if the top-left and
bottom-right indexs are the same, and if top-left is valid. In my case,
topLeft and bottomRight are not equal, so we drop down to line 3368:
d->updateEditorData(topLeft, bottomRight);, which is
shown in the next code block below. This method does a series of checks
on the indexes and a few other things, and then it initiates the change
itself on line 4342 via the call to
delegate->setEditorData(editor, index):
4320 void QAbstractItemViewPrivate::updateEditorData(const QModelIndex &tl, const QModelIndex &br)
4321 {
4322 // we are counting on having relatively few editors
4323 const bool checkIndexes = tl.isValid() && br.isValid();
4324 const QModelIndex parent = tl.parent();
4325 // QTBUG-25370: We need to copy the indexEditorHash, because while we're
4326 // iterating over it, we are calling methods which can allow user code to
4327 // call a method on *this which can modify the member indexEditorHash.
4328 const QIndexEditorHash indexEditorHashCopy = indexEditorHash;
4329 QIndexEditorHash::const_iterator it = indexEditorHashCopy.constBegin();
4330 for (; it != indexEditorHashCopy.constEnd(); ++it) {
4331 QWidget *editor = it.value().widget.data();
4332 const QModelIndex index = it.key();
4333 if (it.value().isStatic || !editor || !index.isValid() ||
4334 (checkIndexes
4335 && (index.row() < tl.row() || index.row() > br.row()
4336 || index.column() < tl.column() || index.column() > br.column()
4337 || index.parent() != parent)))
4338 continue;
4339
4340 QAbstractItemDelegate *delegate = delegateForIndex(index);
4341 if (delegate) {
4342 delegate->setEditorData(editor, index);
4343 }
4344 }
4345 }This is dense. An iterator is being created that generates
QModelIndexes from top-left to bottom-right inclusive, runs
state and bounds checks on them, and if all checks pass, calls
delegate->setEditorData() to get the work done. My
future debugging efforts would most likely begin here, in the
densely-packed check statement at line 4333.
QSortFilterProxyModel
The proxy model is a QSortFilterProxyModel
object. Unlike for the QTableView object above, in this
case the data-changed slot is implemented as a private method within
this class: _q_sourceDataChanged(). It also takes a
top-left and bottom-right pair of QModelIndex objects, and
additionally, a role object, which we don’t need to deal
with here.
The data-change method is copied below. While it is dense and long,
one difference jumps out: at line 1455, there is a different mechanism
for constraining the for loop: an end variable is
initialized by getting the data model’s row count and subtracting one,
and the for loop iterates to that integer value as its upper limit.
Going back to updateEditorData() above, the for loop in the
QAbstractItemView is instead constrained by a
QIndexEditorHash, and there is no ‘length minus one’
accounting.
I suspect that these two methods hold the key to fixing the
discrepancy between how the QAbstractItemView and the
QSortFilterProxyModel handle the off-by-one error
discovered by both myself and DerManu. For the moment, I’ll have to let
this particular puzzle simmer.
1408 void QSortFilterProxyModelPrivate::_q_sourceDataChanged(const QModelIndex &source_top_left,
1409 const QModelIndex &source_bottom_right,
1410 const QVector<int> &roles)
1411 {
1412 Q_Q(QSortFilterProxyModel);
1413 if (!source_top_left.isValid() || !source_bottom_right.isValid())
1414 return;
1415
1416 std::vector<QSortFilterProxyModelDataChanged> data_changed_list;
1417 data_changed_list.emplace_back(source_top_left, source_bottom_right);
1418
1419 // Do check parents if the filter role have changed and we are recursive
1420 if (filter_recursive && (roles.isEmpty() || roles.contains(filter_role))) {
1421 QModelIndex source_parent = source_top_left.parent();
1422
1423 while (source_parent.isValid()) {
1424 data_changed_list.emplace_back(source_parent, source_parent);
1425 source_parent = source_parent.parent();
1426 }
1427 }
1428
1429 for (const QSortFilterProxyModelDataChanged &data_changed : data_changed_list) {
1430 const QModelIndex &source_top_left = data_changed.topLeft;
1431 const QModelIndex &source_bottom_right = data_changed.bottomRight;
1432 const QModelIndex source_parent = source_top_left.parent();
1433
1434 bool change_in_unmapped_parent = false;
1435 IndexMap::const_iterator it = source_index_mapping.constFind(source_parent);
1436 if (it == source_index_mapping.constEnd()) {
1437 // We don't have mapping for this index, so we cannot know how things
1438 // changed (in case the change affects filtering) in order to forward
1439 // the change correctly.
1440 // But we can at least forward the signal "as is", if the row isn't
1441 // filtered out, this is better than nothing.
1442 it = create_mapping_recursive(source_parent);
1443 if (it == source_index_mapping.constEnd())
1444 continue;
1445 change_in_unmapped_parent = true;
1446 }
1447
1448 Mapping *m = it.value();
1449
1450 // Figure out how the source changes affect us
1451 QVector<int> source_rows_remove;
1452 QVector<int> source_rows_insert;
1453 QVector<int> source_rows_change;
1454 QVector<int> source_rows_resort;
1455 int end = qMin(source_bottom_right.row(), m->proxy_rows.count() - 1);
1456 for (int source_row = source_top_left.row(); source_row <= end; ++source_row) {
1457 if (dynamic_sortfilter && !change_in_unmapped_parent) {
1458 if (m->proxy_rows.at(source_row) != -1) {
1459 if (!filterAcceptsRowInternal(source_row, source_parent)) {
1460 // This source row no longer satisfies the filter, so it must be removed
1461 source_rows_remove.append(source_row);
1462 } else if (source_sort_column >= source_top_left.column() && source_sort_column <= source_bottom_right.column()) {
1463 // This source row has changed in a way that may affect sorted order
1464 source_rows_resort.append(source_row);
1465 } else {
1466 // This row has simply changed, without affecting filtering nor sorting
1467 source_rows_change.append(source_row);
1468 }
1469 } else {
1470 if (!itemsBeingRemoved.contains(source_parent, source_row) && filterAcceptsRowInternal(source_row, source_parent)) {
1471 // This source row now satisfies the filter, so it must be added
1472 source_rows_insert.append(source_row);
1473 }
1474 }
1475 } else {
1476 if (m->proxy_rows.at(source_row) != -1)
1477 source_rows_change.append(source_row);
1478 }
1479 }
1480
1481 if (!source_rows_remove.isEmpty()) {
1482 remove_source_items(m->proxy_rows, m->source_rows,
1483 source_rows_remove, source_parent, Qt::Vertical);
1484 QSet<int> source_rows_remove_set = qVectorToSet(source_rows_remove);
1485 QVector<QModelIndex>::iterator childIt = m->mapped_children.end();
1486 while (childIt != m->mapped_children.begin()) {
1487 --childIt;
1488 const QModelIndex source_child_index = *childIt;
1489 if (source_rows_remove_set.contains(source_child_index.row())) {
1490 childIt = m->mapped_children.erase(childIt);
1491 remove_from_mapping(source_child_index);
1492 }
1493 }
1494 }
1495
1496 if (!source_rows_resort.isEmpty()) {
1497 if (needsReorder(source_rows_resort, source_parent)) {
1498 // Re-sort the rows of this level
1499 QList<QPersistentModelIndex> parents;
1500 parents << q->mapFromSource(source_parent);
1501 emit q->layoutAboutToBeChanged(parents, QAbstractItemModel::VerticalSortHint);
1502 QModelIndexPairList source_indexes = store_persistent_indexes();
1503 remove_source_items(m->proxy_rows, m->source_rows, source_rows_resort,
1504 source_parent, Qt::Vertical, false);
1505 sort_source_rows(source_rows_resort, source_parent);
1506 insert_source_items(m->proxy_rows, m->source_rows, source_rows_resort,
1507 source_parent, Qt::Vertical, false);
1508 update_persistent_indexes(source_indexes);
1509 emit q->layoutChanged(parents, QAbstractItemModel::VerticalSortHint);
1510 }
1511 // Make sure we also emit dataChanged for the rows
1512 source_rows_change += source_rows_resort;
1513 }
1514
1515 if (!source_rows_change.isEmpty()) {
1516 // Find the proxy row range
1517 int proxy_start_row;
1518 int proxy_end_row;
1519 proxy_item_range(m->proxy_rows, source_rows_change,
1520 proxy_start_row, proxy_end_row);
1521 // ### Find the proxy column range also
1522 if (proxy_end_row >= 0) {
1523 // the row was accepted, but some columns might still be filtered out
1524 int source_left_column = source_top_left.column();
1525 while (source_left_column < source_bottom_right.column()
1526 && m->proxy_columns.at(source_left_column) == -1)
1527 ++source_left_column;
1528 if (m->proxy_columns.at(source_left_column) != -1) {
1529 const QModelIndex proxy_top_left = create_index(
1530 proxy_start_row, m->proxy_columns.at(source_left_column), it);
1531 int source_right_column = source_bottom_right.column();
1532 while (source_right_column > source_top_left.column()
1533 && m->proxy_columns.at(source_right_column) == -1)
1534 --source_right_column;
1535 if (m->proxy_columns.at(source_right_column) != -1) {
1536 const QModelIndex proxy_bottom_right = create_index(
1537 proxy_end_row, m->proxy_columns.at(source_right_column), it);
1538 emit q->dataChanged(proxy_top_left, proxy_bottom_right, roles);
1539 }
1540 }
1541 }
1542 }
1543
1544 if (!source_rows_insert.isEmpty()) {
1545 sort_source_rows(source_rows_insert, source_parent);
1546 insert_source_items(m->proxy_rows, m->source_rows,
1547 source_rows_insert, source_parent, Qt::Vertical);
1548 }
1549 }
1550 }Thank you for reading!